Nov 03 02:58:10.999910 ip-10-0-138-11 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 02:58:10.999922 ip-10-0-138-11 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 02:58:10.999929 ip-10-0-138-11 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 02:58:11.000173 ip-10-0-138-11 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 02:58:21.003579 ip-10-0-138-11 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 02:58:21.003597 ip-10-0-138-11 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4e5ef4395727455e8fc36014b90deb7e --
Nov 03 03:00:30.465857 ip-10-0-138-11 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 03:00:30.837498 ip-10-0-138-11 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 03:00:30.837498 ip-10-0-138-11 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 03:00:30.837498 ip-10-0-138-11 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 03:00:30.837498 ip-10-0-138-11 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 03:00:30.840916 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840839    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.840916 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840917    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840922    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840925    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840928    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840931    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840934    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840936    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840939    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840945    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840948    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840951    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840954    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840956    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840959    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840961    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840964    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840966    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840969    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840971    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840974    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.840983 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840977    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840979    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840982    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840984    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840987    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840990    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840993    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840996    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.840999    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841001    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841004    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841006    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841009    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841011    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841013    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841016    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841018    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841021    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841027    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841030    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.841443 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841032    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841034    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841038    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841040    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841043    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841045    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841047    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841050    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841052    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841054    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841057    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841059    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841062    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841064    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841067    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841070    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841072    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841075    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841077    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841080    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.842166 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841082    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841085    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841087    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841090    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841092    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841094    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841097    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841099    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841102    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841106    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841110    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841113    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841116    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841119    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841121    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841124    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841126    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841129    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841131    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.842702 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841134    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841136    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841138    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841141    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841143    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841146    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841149    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841151    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841154    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841156    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841159    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841161    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841164    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841166    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841168    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841171    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841173    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841177    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841181    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.843147 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841183    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841186    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841188    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841191    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841194    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841197    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.841200    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842070    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842078    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842082    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842085    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842088    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842091    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842094    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842096    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842099    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842101    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842104    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842106    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842109    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.843603 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842112    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842114    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842117    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842120    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842122    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842127    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842130    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842133    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842136    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842139    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842141    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842144    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842146    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842149    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842151    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842155    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842159    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842162    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842165    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.844187 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842168    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842171    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842174    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842177    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842180    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842182    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842185    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842188    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842190    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842192    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842195    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842197    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842201    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842204    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842207    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842209    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842212    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842214    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842217    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842219    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842221    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.844646 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842224    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842226    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842229    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842232    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842234    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842237    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842239    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842241    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842244    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842246    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842248    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842251    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842254    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842256    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842258    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842261    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842264    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842266    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842269    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.845144 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842271    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842274    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842276    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842278    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842282    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842284    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842287    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842290    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842292    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842294    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842297    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842300    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842303    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842305    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842308    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842310    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842313    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842315    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842318    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842320    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.845637 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842322    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842325    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842327    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842330    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842332    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842335    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842337    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842340    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842342    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842345    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842347    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842350    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842352    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842355    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842836    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842841    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842845    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842848    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842851    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842853    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.846123 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842856    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842859    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842862    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842864    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842867    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842869    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842872    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842874    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842877    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842879    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842882    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842884    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842887    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842889    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842891    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842894    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842897    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842899    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842902    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842905    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.846622 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842907    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842910    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842912    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842915    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842917    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842920    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842922    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842925    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842929    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842933    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842937    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842941    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842944    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842946    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842949    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842951    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842954    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842956    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842959    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.847093 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842961    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842964    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842966    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842969    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842971    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842974    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842976    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842978    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842981    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842983    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842986    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842988    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842991    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842994    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842996    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.842999    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843001    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843003    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843006    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843008    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.847568 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843011    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843013    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843016    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843018    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843021    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843023    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843026    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843028    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843031    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843033    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843035    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843038    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843040    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843044    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843047    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843050    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843052    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843055    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843057    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.848047 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843059    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843062    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843064    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843067    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843071    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843074    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843077    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843079    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843083    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843085    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843088    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843090    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843092    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843095    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843099    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843101    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843104    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843106    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843109    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843111    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.848474 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843114    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843116    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843204    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843210    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843216    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843220    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843225    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843228    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843232    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843236    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843239    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843242    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843245    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843248    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843251    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843254    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843259    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843262    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843264    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843268    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843273    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843276    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843279    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843282    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 03:00:30.848950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843285    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843289    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843294    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843297    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843300    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843303    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843306    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843309    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843312    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843314    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843319    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843321    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843324    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843327    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843330    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843333    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843337    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843340    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843343    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843346    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843349    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843353    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843356    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843358    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843363    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 03:00:30.849505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843366    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843369    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843372    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843375    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843378    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843381    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843384    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843388    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843391    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843396    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843399    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843402    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843405    2576 flags.go:64] FLAG: --help="false"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843408    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843411    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843414    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843417    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843420    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843423    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843426    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843429    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843431    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843434    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 03:00:30.850157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843437    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843440    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843443    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843446    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843448    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843451    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843454    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843457    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843460    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843463    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843466    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843472    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843475    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843478    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843481    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843484    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843487    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843490    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843494    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843498    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843501    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843505    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843508    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843511    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843514    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 03:00:30.850780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843517    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843520    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843523    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843525    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843532    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843535    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843538    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843541    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843561    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843564    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843567    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843570    2576 flags.go:64] FLAG: --port="10250"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843572    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843575    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02cf534ab24e44e9e"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843578    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843583    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843585    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843588    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843593    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843596    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843598    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843602    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843605    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843609    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843612    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843616    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 03:00:30.851620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843618    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843621    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843625    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843628    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843630    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843633    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843637    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843640    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843643    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843645    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843648    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843651    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843654    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843657    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843660    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843662    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843668    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843671    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843673    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843678    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843680    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843684    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843687    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843690    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843692    2576 flags.go:64] FLAG: --v="2"
Nov 03 03:00:30.852234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843697    2576 flags.go:64] FLAG: --version="false"
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843701    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843705    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.843708    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843838    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843841    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843845    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843848    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843850    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843853    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843855    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843858    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843860    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843863    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843865    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843868    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843870    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843873    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843876    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843879    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.852893 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843881    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843883    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843886    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843888    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843891    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843893    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843896    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843900    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843904    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843907    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843909    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843912    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843914    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843917    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843919    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843922    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843925    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843927    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843931    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.853360 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843933    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843936    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843938    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843941    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843943    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843946    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843948    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843950    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843953    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843955    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843958    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843960    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843963    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843965    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843968    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843970    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843973    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843975    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843977    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.853839 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843980    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843982    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843986    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843989    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843993    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843995    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.843998    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844001    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844004    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844006    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844009    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844012    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844015    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844018    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844021    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844023    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844026    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844028    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844031    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.854278 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844033    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844036    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844038    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844041    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844044    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844046    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844049    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844051    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844054    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844056    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844059    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844062    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844064    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844066    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844069    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844073    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844075    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844078    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844080    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.854789 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844083    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844085    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844087    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844090    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844092    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844094    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844099    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844102    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844104    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844107    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844109    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844112    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844114    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844117    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.844141    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844177    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.855234 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844181    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844185    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844187    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844190    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844193    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844199    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844202    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844204    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844207    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844209    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844212    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844216    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844218    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844222    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844226    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844229    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844231    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844234    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.855627 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844237    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844240    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844242    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844244    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844248    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844251    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844253    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844256    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844258    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844261    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844263    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844265    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844268    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844270    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844273    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844276    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844278    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844280    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844283    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.856105 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844286    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844288    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844291    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844293    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844295    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844298    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844302    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844305    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844307    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844310    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844314    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844317    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844320    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844324    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844326    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844329    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844331    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844335    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844338    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.856611 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844341    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844343    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844346    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844348    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844350    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844353    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844356    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844358    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844360    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844363    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844365    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844368    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844370    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844374    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844376    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844378    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844381    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844383    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844386    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.857057 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844389    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844392    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844394    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844396    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844399    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844401    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844403    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844406    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844408    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844411    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844413    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844417    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844420    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844423    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844425    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844428    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844430    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844433    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844435    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844437    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.857505 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844440    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844442    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844445    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844447    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844449    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844452    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844454    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844457    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.844459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.851637    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.851650    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.851665    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.851671    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851731    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851737    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851740    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851743    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851746    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851749    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851752    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.858000 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851754    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851757    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851760    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851763    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851765    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851768    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851771    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851774    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851776    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851778    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851781    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851783    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851786    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851788    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851791    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851793    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851795    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851798    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851800    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851803    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.858459 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851805    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851808    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851810    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851813    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851815    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851818    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851820    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851823    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851825    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851827    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851830    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851832    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851834    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851837    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851839    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851842    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851844    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851846    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851849    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.859037 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851852    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851855    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851857    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851859    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851862    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851864    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851867    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851869    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851871    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851874    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851876    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851879    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851881    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851884    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851886    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851888    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851891    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851893    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851896    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.859855 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851898    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851901    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851903    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851906    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851908    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851910    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851914    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851918    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851921    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851924    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851927    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851929    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851932    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851935    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851937    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851939    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851942    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851945    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851949    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.860336 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851952    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851954    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851957    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851959    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851961    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851964    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851966    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851968    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851971    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851973    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851976    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851978    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851980    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851983    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851986    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851988    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851990    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851993    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851995    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.851997    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.860848 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852000    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852003    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.852029    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852172    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852178    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852181    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852184    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852187    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852190    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852193    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852195    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852197    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852200    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852202    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852204    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.861318 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852207    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852209    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852212    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852214    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852217    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852219    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852221    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852224    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852226    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852229    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852231    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852234    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852236    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852239    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852241    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852244    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852246    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852249    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852252    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.861761 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852254    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852257    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852259    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852262    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852264    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852267    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852269    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852272    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852274    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852276    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852279    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852281    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852284    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852286    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852289    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852291    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852293    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852296    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852298    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852301    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.862199 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852303    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852306    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852308    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852310    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852313    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852315    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852317    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852320    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852322    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852325    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852327    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852329    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852334    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852336    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852339    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852343    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852347    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852350    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852353    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.862686 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852357    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852360    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852362    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852367    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852369    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852372    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852374    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852377    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852379    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852382    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852384    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852387    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852389    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852392    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852394    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852397    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852399    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852401    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852404    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852406    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.863125 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852409    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852411    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852414    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852416    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852419    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852422    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852425    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852427    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852430    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852432    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852434    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852437    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852439    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852442    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852445    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852448    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.863633 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.852470    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852508    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852512    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852516    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852519    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852522    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852524    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852527    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852529    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852532    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852534    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852537    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852539    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852542    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852559    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852563    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852567    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852570    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852572    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852575    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:00:30.864029 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852577    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852580    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852584    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852586    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852589    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852592    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852594    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852597    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852599    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852602    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852605    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852608    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852611    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852614    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852616    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852619    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852621    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852624    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852626    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852629    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:00:30.864481 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852632    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852634    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852637    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852639    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852642    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852644    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852647    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852649    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852652    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852654    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852657    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852659    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852661    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852664    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852670    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852673    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852677    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852679    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852682    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:00:30.865026 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852684    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852687    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852690    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852692    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852696    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852699    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852701    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852704    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852706    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852709    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852711    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852714    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852716    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852718    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852721    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852723    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852726    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852728    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852731    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:00:30.865454 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852733    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852736    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852738    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852741    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852743    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852746    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852748    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852751    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852754    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852757    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852760    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852763    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852765    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852767    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852770    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852772    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852775    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852779    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852781    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:00:30.865919 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852787    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852789    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852792    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852794    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852797    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852799    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852801    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852805    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:30.852807    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.852855    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.855639    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.856572    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.858790    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.858963    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.862255    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 03:00:30.866378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.863482    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 03:00:30.873324 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.873309    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 03:00:30.876283 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.876269    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 03:00:30.877038 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.877005    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 03:00:30.881462 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.881445    2576 fs.go:135] Filesystem UUIDs: map[3814681a-6274-4549-bd24-b3a06cd3952b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c903799d-0ba2-4815-bce0-8348f9bc9138:/dev/nvme0n1p3]
Nov 03 03:00:30.881527 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.881463    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 03:00:30.886811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.886698    2576 manager.go:217] Machine: {Timestamp:2026-11-03 03:00:30.885533556 +0000 UTC m=+0.312266649 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098527 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e34d048943c3fb7c22ff26aee64b9 SystemUUID:ec2e34d0-4894-3c3f-b7c2-2ff26aee64b9 BootID:4e5ef439-5727-455e-8fc3-6014b90deb7e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:8c:80:ab:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:8c:80:ab:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ab:20:4b:51:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 03:00:30.886811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.886810    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 03:00:30.886938 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.886912    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 03:00:30.888892 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.888871    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 03:00:30.889026 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.888894    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-11.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 03:00:30.889073 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.889028    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 03:00:30.889073 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.889036    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 03:00:30.889073 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.889065    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 03:00:30.889147 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.889083    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 03:00:30.889738 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.889728    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 03:00:30.893389 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.893376    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 03:00:30.893517 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.893508    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 03:00:30.894131 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.894119    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 03:00:30.894160 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.894137    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 03:00:30.894160 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.894147    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 03:00:30.894216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.894193    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 03:00:30.894216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.894205    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 03:00:30.894615 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.894600    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 03:00:30.896757 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.896740    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 03:00:30.902839 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.902822    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 03:00:30.903596 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903579    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 03:00:30.903680 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903611    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 03:00:30.903772 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903762    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 03:00:30.903818 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903778    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 03:00:30.903818 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903784    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 03:00:30.903818 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903790    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 03:00:30.903818 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903795    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 03:00:30.903818 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.903801    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 03:00:30.904331 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904321    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 03:00:30.904362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904333    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 03:00:30.904362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904340    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 03:00:30.904362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904346    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 03:00:30.904362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904354    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 03:00:30.904362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904363    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 03:00:30.904994 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904985    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 03:00:30.905031 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.904996    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 03:00:30.908535 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.908523    2576 server.go:1263] "Started kubelet"
Nov 03 03:00:30.908663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.908614    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 03:00:30.908728 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.908642    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 03:00:30.908728 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.908713    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 03:00:30.909293 ip-10-0-138-11 systemd[1]: Started Kubernetes Kubelet.
Nov 03 03:00:30.910335 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.910322    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 03:00:30.913041 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.913024    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 03:00:30.914115 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:30.914094    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 03:00:30.917143 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.917122    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 03:00:30.917591 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.917571    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 03:00:30.917842 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:30.917815    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:30.918299 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918283    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 03:00:30.918299 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918286    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 03:00:30.918402 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918311    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 03:00:30.918456 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918444    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 03:00:30.918496 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918458    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 03:00:30.918975 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918952    2576 factory.go:55] Registering systemd factory
Nov 03 03:00:30.918975 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.918967    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 03:00:30.919193 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.919181    2576 factory.go:153] Registering CRI-O factory
Nov 03 03:00:30.919257 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.919195    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 03:00:30.919257 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.919246    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 03:00:30.919257 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.919259    2576 factory.go:103] Registering Raw factory
Nov 03 03:00:30.919372 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.919268    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 03:00:30.920280 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.920264    2576 manager.go:319] Starting recovery of all containers
Nov 03 03:00:30.930649 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.930630    2576 manager.go:324] Recovery completed
Nov 03 03:00:30.935206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.935192    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:30.936772 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.936756    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:30.936850 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.936787    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:30.936850 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.936797    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:30.937294 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.937280    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 03:00:30.937294 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.937290    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 03:00:30.937394 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.937299    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:30.937394 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.937306    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 03:00:30.937989 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.937975    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:30.938048 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.938000    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:30.938048 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.938013    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:30.939573 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.939562    2576 policy_none.go:50] "Start"
Nov 03 03:00:30.939613 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.939577    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 03:00:30.939613 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.939586    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 03:00:30.941663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.941643    2576 policy_none.go:44] "Start"
Nov 03 03:00:30.976064 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:30.975922    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 03:00:31.001608 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.001595    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 03:00:31.001698 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.001634    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 03:00:31.001698 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.001648    2576 server.go:87] "Starting device plugin registration server"
Nov 03 03:00:31.001890 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.001877    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 03:00:31.001936 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.001895    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 03:00:31.002161 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.002017    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 03:00:31.002161 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.002123    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 03:00:31.002161 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.002131    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 03:00:31.006062 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.006045    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 03:00:31.006161 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.006093    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:31.049704 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.049673    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 03:00:31.051162 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.051001    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 03:00:31.051162 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.051031    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 03:00:31.051162 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.051053    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 03:00:31.051162 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.051119    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 03:00:31.102249 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.102183    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:31.103106 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.103084    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:31.103224 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.103124    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:31.103224 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.103135    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:31.103224 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.103160    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.109344 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.109327    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-11.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 03:00:31.109440 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.109381    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-11.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 03:00:31.110122 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.109336    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-11.us-west-1.compute.internal.18e469c162dc15f0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-11.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-11.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:30.908495344 +0000 UTC m=+0.335228438,LastTimestamp:2026-11-03 03:00:30.908495344 +0000 UTC m=+0.335228438,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-11.us-west-1.compute.internal,}"
Nov 03 03:00:31.151492 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.151455    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"]
Nov 03 03:00:31.151600 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.151539    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:31.152367 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.152352    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:31.152420 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.152380    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:31.152420 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.152391    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:31.154318 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.154307    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:31.154488 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.154471    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.154570 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.154510    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:31.155035 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.155017    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:31.155117 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.155051    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:31.155117 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.155062    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:31.155117 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.155022    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:31.155208 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.155136    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:31.155208 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.155149    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:31.156829 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.156813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.156894 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.156845    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:31.157564 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.157534    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:31.157632 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.157580    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:31.157632 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.157594    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:31.163626 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.163610    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 03:00:31.188149 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.188128    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.188217 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.188150    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lwd6z"
Nov 03 03:00:31.189795 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.189781    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.198486 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.198470    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lwd6z"
Nov 03 03:00:31.219379 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.219358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.219473 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.219333    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-11.us-west-1.compute.internal.18e469c1648b917e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-11.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-11.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-11.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:30.93677299 +0000 UTC m=+0.363506084,LastTimestamp:2026-11-03 03:00:30.93677299 +0000 UTC m=+0.363506084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-11.us-west-1.compute.internal,}"
Nov 03 03:00:31.219473 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.219388    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 03:00:31.219473 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.219386    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-11.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 03:00:31.219609 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.219382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.219609 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.219510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8452f8e9cc16a665620f2f318ca698f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"8452f8e9cc16a665620f2f318ca698f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.219669 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.219611    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 03:00:31.273204 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.273180    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:00:31.273879 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.273863    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-11.us-west-1.compute.internal\" not found" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.320559 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.320520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.320663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.320561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.320663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.320587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8452f8e9cc16a665620f2f318ca698f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"8452f8e9cc16a665620f2f318ca698f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.320663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.320627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8452f8e9cc16a665620f2f318ca698f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"8452f8e9cc16a665620f2f318ca698f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.320663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.320632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.320663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.320627    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.413665 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.413598    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-11.us-west-1.compute.internal\" not found" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.474974 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.474946    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:31.475980 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.475951    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:31.476061 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.475984    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:31.476061 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.475994    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:31.476061 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.476021    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.491147 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.491128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.493263 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.493248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.539074 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.539046    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:31.539074 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.539076    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-11.us-west-1.compute.internal\": node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:31.661647 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.661618    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:31.762350 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.762271    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:31.856623 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.856591    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 03:00:31.857142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:31.856783    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 03:00:31.857142 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.856788    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a058e290254ff44d893eba1331bf9de6-33cafa47f6ec0d31.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-11.us-west-1.compute.internal.18e469c1648bfdfe\": read tcp 10.0.138.11:36104->18.221.54.233:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-11.us-west-1.compute.internal.18e469c1648bfdfe  default   5056 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-11.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-11.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-11.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 03:00:30 +0000 UTC,LastTimestamp:2026-11-03 03:00:31.103138987 +0000 UTC m=+0.529872081,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-11.us-west-1.compute.internal,}"
Nov 03 03:00:31.862828 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.862793    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:31.963525 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:31.963479    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.064040 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.063974    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.154616 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.154585    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:00:32.155411 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.155392    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:00:32.155514 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.155426    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:00:32.155514 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.155437    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:00:32.164142 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.164126    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.202811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.202764    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 02:55:32 +0000 UTC" deadline="2028-07-21 06:41:42.582651156 +0000 UTC"
Nov 03 03:00:32.202811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.202805    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15027h41m10.37984915s"
Nov 03 03:00:32.264814 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.264785    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.277536 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.277508    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:00:32.365639 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.365613    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.373160 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.373139    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:00:32.466706 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.466673    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.567749 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.567695    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.585242 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:32.585204    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8452f8e9cc16a665620f2f318ca698f0.slice/crio-6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc WatchSource:0}: Error finding container 6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc: Status 404 returned error can't find the container with id 6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc
Nov 03 03:00:32.585462 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:32.585440    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a743d363639b28a6d7229f5e9636d9.slice/crio-3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b WatchSource:0}: Error finding container 3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b: Status 404 returned error can't find the container with id 3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b
Nov 03 03:00:32.590406 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.590391    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:00:32.668060 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:32.667992    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:00:32.718283 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.718259    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:00:32.818906 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.818875    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:32.886861 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.886830    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 03:00:32.888120 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.888100    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:00:32.895649 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.895633    2576 apiserver.go:52] "Watching apiserver"
Nov 03 03:00:32.919107 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.919045    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 03:00:32.953386 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:32.953363    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 03:00:33.007283 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.007257    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 03:00:33.007589 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.007568    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5xflz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4","openshift-cluster-node-tuning-operator/tuned-hgh8q","openshift-dns/node-resolver-77z26","openshift-image-registry/node-ca-4msjq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-568md","openshift-network-operator/iptables-alerter-gmpwj","openshift-ovn-kubernetes/ovnkube-node-cvkw5","kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-rb9rv","openshift-multus/multus-jb7sk","openshift-multus/network-metrics-daemon-6k7g2","kube-system/global-pull-secret-syncer-rqm5c"]
Nov 03 03:00:33.011713 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.011696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:33.011790 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.011771    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:33.013919 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.013904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.013975 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.013964    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:33.016212 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.016195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.018421 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.018403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.020698 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.020677    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.023056 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.023037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.023131 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.023114    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:33.025640 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.025617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.028537 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.028520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.030372 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81335b4a-183a-49d7-81d0-8bff20d94e77-tmp-dir\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.030449 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.030449 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-modprobe-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030567 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64651a3d-f058-40c3-9171-53ba4e35d04a-serviceca\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.030567 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:33.030682 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysconfig\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030682 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-systemd\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030682 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-run\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030682 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-sys\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030682 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-var-lib-kubelet\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-tuned\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81335b4a-183a-49d7-81d0-8bff20d94e77-hosts-file\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64651a3d-f058-40c3-9171-53ba4e35d04a-host\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skpl\" (UniqueName: \"kubernetes.io/projected/64651a3d-f058-40c3-9171-53ba4e35d04a-kube-api-access-5skpl\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-host\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.030871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdb6f\" (UniqueName: \"kubernetes.io/projected/794ed6bd-5ce9-4aea-99ab-75607b2cd846-kube-api-access-rdb6f\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4f4\" (UniqueName: \"kubernetes.io/projected/81335b4a-183a-49d7-81d0-8bff20d94e77-kube-api-access-kx4f4\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgk8\" (UniqueName: \"kubernetes.io/projected/996d2ead-9513-41c2-acd6-1ced1f624131-kube-api-access-xdgk8\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.030991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-kubernetes\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.031019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-conf\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.031044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-lib-modules\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.031066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-tmp\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.031206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.031206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.033735 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.033719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.036415 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.036399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.039014 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.038995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.055185 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.055130    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"8452f8e9cc16a665620f2f318ca698f0","Type":"ContainerStarted","Data":"6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc"}
Nov 03 03:00:33.056381 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.056358    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"f0a743d363639b28a6d7229f5e9636d9","Type":"ContainerStarted","Data":"3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b"}
Nov 03 03:00:33.075216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.075193    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.075305 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.075197    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.075364 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.075314    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgzd9\""
Nov 03 03:00:33.075364 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.075318    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.075364 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.075342    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcpl5\""
Nov 03 03:00:33.075838 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.075820    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.076897 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.076876    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.077056 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.077026    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.077214 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.077195    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 03:00:33.077705 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.077692    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrqj7\""
Nov 03 03:00:33.119661 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.119613    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 03:00:33.124717 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.124698    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 03:00:33.124802 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.124770    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7lz87"
Nov 03 03:00:33.132082 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-modprobe-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132150 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-multus\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.132150 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-ovn\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.132150 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-log-socket\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.132280 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbdq\" (UniqueName: \"kubernetes.io/projected/a3c603e8-5a0a-42af-a866-3439e1fdef56-kube-api-access-xhbdq\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.132280 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132242    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-modprobe-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132280 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-binary-copy\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.132374 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-sys\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132374 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdb6f\" (UniqueName: \"kubernetes.io/projected/794ed6bd-5ce9-4aea-99ab-75607b2cd846-kube-api-access-rdb6f\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132435 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-sys\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132435 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-device-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.132517 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.132580 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-multus-certs\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.132580 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-konnectivity-ca\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.132715 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81335b4a-183a-49d7-81d0-8bff20d94e77-hosts-file\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.132715 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64651a3d-f058-40c3-9171-53ba4e35d04a-host\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.132715 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132715 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81335b4a-183a-49d7-81d0-8bff20d94e77-hosts-file\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.132715 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-registration-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132740    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64651a3d-f058-40c3-9171-53ba4e35d04a-host\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-kubelet\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-multus-daemon-config\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-dbus\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.132946 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-kubernetes\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-kubernetes\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.132983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-lib-modules\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxb44\" (UniqueName: \"kubernetes.io/projected/7a2050bd-da88-4923-b686-4c9e70226b59-kube-api-access-xxb44\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-cni-binary-copy\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133068    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-lib-modules\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9dm\" (UniqueName: \"kubernetes.io/projected/44942777-b4b6-4264-aeff-7c88943cf14e-kube-api-access-7v9dm\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-agent-certs\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cnibin\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-bin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-netd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.133252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovn-node-metrics-cert\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysconfig\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-tuned\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysconfig\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2050bd-da88-4923-b686-4c9e70226b59-host-slash\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-cnibin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5vf\" (UniqueName: \"kubernetes.io/projected/f09cc1ed-086f-4b12-a9ff-e465fbe94038-kube-api-access-8d5vf\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-socket-dir-parent\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-system-cni-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-conf\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-k8s-cni-cncf-io\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133662    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-kubelet\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133723    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-conf\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-hostroot\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-conf-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.133792 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.133774    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.133832    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:33.633815791 +0000 UTC m=+3.060548895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-etc-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-netns\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-config\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-script-lib\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-systemd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.133990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-var-lib-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-env-overrides\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4f4\" (UniqueName: \"kubernetes.io/projected/81335b4a-183a-49d7-81d0-8bff20d94e77-kube-api-access-kx4f4\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-tmp\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.134442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81335b4a-183a-49d7-81d0-8bff20d94e77-tmp-dir\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-system-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-systemd-units\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64651a3d-f058-40c3-9171-53ba4e35d04a-serviceca\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-systemd\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-run\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-var-lib-kubelet\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81335b4a-183a-49d7-81d0-8bff20d94e77-tmp-dir\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-etc-selinux\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-sys-fs\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a2050bd-da88-4923-b686-4c9e70226b59-iptables-alerter-script\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skpl\" (UniqueName: \"kubernetes.io/projected/64651a3d-f058-40c3-9171-53ba4e35d04a-kube-api-access-5skpl\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-run\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-host\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-socket-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134690    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-systemd\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-etc-kubernetes\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-var-lib-kubelet\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-kubelet-config\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-slash\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-host\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-node-log\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64651a3d-f058-40c3-9171-53ba4e35d04a-serviceca\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgk8\" (UniqueName: \"kubernetes.io/projected/996d2ead-9513-41c2-acd6-1ced1f624131-kube-api-access-xdgk8\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.134921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9kdd\" (UniqueName: \"kubernetes.io/projected/ff3b7b52-b0ad-4965-ba72-39330944d766-kube-api-access-v9kdd\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.135030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-netns\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.135733 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.135061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-bin\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.137273 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.137255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-tmp\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.137318 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.137284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-tuned\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.184139 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184067    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.184139 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184130    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 03:00:33.184139 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184130    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.184309 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184137    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 03:00:33.184400 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184381    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.184477 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184445    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.184706 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184663    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 03:00:33.184706 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184680    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.184706 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184688    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.184960 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184719    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 03:00:33.184960 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184883    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 03:00:33.184960 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184892    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 03:00:33.184960 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184897    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.184960 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184924    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 03:00:33.184960 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184924    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.185218 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.184974    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 03:00:33.185218 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.185024    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 03:00:33.186030 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186016    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp5b4\""
Nov 03 03:00:33.186109 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186064    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fg6rm\""
Nov 03 03:00:33.186109 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186104    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqjzv\""
Nov 03 03:00:33.186300 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186283    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 03:00:33.186365 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186317    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rnqls\""
Nov 03 03:00:33.186365 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186354    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 03:00:33.186546 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186533    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 03:00:33.186657 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186600    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tvk4\""
Nov 03 03:00:33.186657 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.186609    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d2r77\""
Nov 03 03:00:33.200708 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.200680    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4f4\" (UniqueName: \"kubernetes.io/projected/81335b4a-183a-49d7-81d0-8bff20d94e77-kube-api-access-kx4f4\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.200708 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.200691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdb6f\" (UniqueName: \"kubernetes.io/projected/794ed6bd-5ce9-4aea-99ab-75607b2cd846-kube-api-access-rdb6f\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.204514 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.204499    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:00:33.204569 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.204517    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:00:33.204569 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.204527    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:33.204658 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.204584    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:33.704570196 +0000 UTC m=+3.131303279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:33.206653 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.206620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skpl\" (UniqueName: \"kubernetes.io/projected/64651a3d-f058-40c3-9171-53ba4e35d04a-kube-api-access-5skpl\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.207310 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.207288    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgk8\" (UniqueName: \"kubernetes.io/projected/996d2ead-9513-41c2-acd6-1ced1f624131-kube-api-access-xdgk8\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.235601 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-registration-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.235601 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-kubelet\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.235748 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-multus-daemon-config\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.235748 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-dbus\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.235748 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-registration-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.235748 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-kubelet\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.235748 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxb44\" (UniqueName: \"kubernetes.io/projected/7a2050bd-da88-4923-b686-4c9e70226b59-kube-api-access-xxb44\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-cni-binary-copy\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235796    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-dbus\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9dm\" (UniqueName: \"kubernetes.io/projected/44942777-b4b6-4264-aeff-7c88943cf14e-kube-api-access-7v9dm\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235936    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-agent-certs\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.235988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cnibin\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.236037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-bin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cnibin\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-netd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236119    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-bin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236159    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236163    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-netd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.236471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-cni-binary-copy\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236848 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovn-node-metrics-cert\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.236848 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2050bd-da88-4923-b686-4c9e70226b59-host-slash\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.236848 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-cnibin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236848 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5vf\" (UniqueName: \"kubernetes.io/projected/f09cc1ed-086f-4b12-a9ff-e465fbe94038-kube-api-access-8d5vf\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.236848 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-socket-dir-parent\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.236848 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.236765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2050bd-da88-4923-b686-4c9e70226b59-host-slash\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.237113 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237090    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-cnibin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237167 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-system-cni-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.237222 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237161    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-socket-dir-parent\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237222 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-k8s-cni-cncf-io\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237311 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-kubelet\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.237311 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-multus-daemon-config\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237311 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-hostroot\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237311 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-system-cni-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.237499 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-conf-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237499 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-conf-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237499 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237378    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-kubelet\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.237499 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237426    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-k8s-cni-cncf-io\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237499 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-hostroot\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-etc-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-netns\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.237676    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-config\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-script-lib\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.237754 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.237746    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:33.737719136 +0000 UTC m=+3.164452234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-etc-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-systemd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-var-lib-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237846    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-netns\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-env-overrides\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.237985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.238142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-system-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-systemd-units\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-etc-selinux\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-sys-fs\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a2050bd-da88-4923-b686-4c9e70226b59-iptables-alerter-script\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-socket-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238345    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-script-lib\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-etc-kubernetes\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-kubelet-config\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-slash\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-node-log\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kdd\" (UniqueName: \"kubernetes.io/projected/ff3b7b52-b0ad-4965-ba72-39330944d766-kube-api-access-v9kdd\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-netns\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-bin\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.238597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238576    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-multus\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238620    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-systemd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238663    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-kubelet-config\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238698    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-multus\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238752    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-slash\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-var-lib-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-system-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238813    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-netns\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238838    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-systemd-units\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-bin\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238918    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-etc-selinux\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.238973    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239038    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-sys-fs\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239077    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-config\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-env-overrides\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-socket-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.239293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-node-log\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-ovn\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239297    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-ovn\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-etc-kubernetes\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-log-socket\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbdq\" (UniqueName: \"kubernetes.io/projected/a3c603e8-5a0a-42af-a866-3439e1fdef56-kube-api-access-xhbdq\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-binary-copy\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-device-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-multus-certs\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-konnectivity-ca\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239592    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a2050bd-da88-4923-b686-4c9e70226b59-iptables-alerter-script\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovn-node-metrics-cert\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-device-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239716    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-multus-certs\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.240119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-log-socket\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.240912 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.239992    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-agent-certs\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.240912 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.240179    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-binary-copy\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.240912 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.240235    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-konnectivity-ca\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.303600 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.303569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9dm\" (UniqueName: \"kubernetes.io/projected/44942777-b4b6-4264-aeff-7c88943cf14e-kube-api-access-7v9dm\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.314362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.314330    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxb44\" (UniqueName: \"kubernetes.io/projected/7a2050bd-da88-4923-b686-4c9e70226b59-kube-api-access-xxb44\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.316253 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.316233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5vf\" (UniqueName: \"kubernetes.io/projected/f09cc1ed-086f-4b12-a9ff-e465fbe94038-kube-api-access-8d5vf\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.322178 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.322151    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kdd\" (UniqueName: \"kubernetes.io/projected/ff3b7b52-b0ad-4965-ba72-39330944d766-kube-api-access-v9kdd\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.322903 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.322884    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbdq\" (UniqueName: \"kubernetes.io/projected/a3c603e8-5a0a-42af-a866-3439e1fdef56-kube-api-access-xhbdq\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.327783 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.327767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:00:33.335726 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.335709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77z26"
Nov 03 03:00:33.339100 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.339077    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794ed6bd_5ce9_4aea_99ab_75607b2cd846.slice/crio-979a30f47fb0ac699a76e59d660dfd9691e00ddbb9f8e1a1acaebb60bfbd86e7 WatchSource:0}: Error finding container 979a30f47fb0ac699a76e59d660dfd9691e00ddbb9f8e1a1acaebb60bfbd86e7: Status 404 returned error can't find the container with id 979a30f47fb0ac699a76e59d660dfd9691e00ddbb9f8e1a1acaebb60bfbd86e7
Nov 03 03:00:33.344816 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.344797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:00:33.346010 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.345983    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81335b4a_183a_49d7_81d0_8bff20d94e77.slice/crio-a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d WatchSource:0}: Error finding container a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d: Status 404 returned error can't find the container with id a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d
Nov 03 03:00:33.354025 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.354001    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64651a3d_f058_40c3_9171_53ba4e35d04a.slice/crio-61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1 WatchSource:0}: Error finding container 61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1: Status 404 returned error can't find the container with id 61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1
Nov 03 03:00:33.361522 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.361502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:33.371987 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.371963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:00:33.372141 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.372118    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35125e2e_9a2d_4577_98ca_5966cfe1d3a1.slice/crio-f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f WatchSource:0}: Error finding container f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f: Status 404 returned error can't find the container with id f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f
Nov 03 03:00:33.374137 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.374118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:00:33.377003 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.376984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:00:33.381226 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.381204    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3b7b52_b0ad_4965_ba72_39330944d766.slice/crio-7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338 WatchSource:0}: Error finding container 7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338: Status 404 returned error can't find the container with id 7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338
Nov 03 03:00:33.384332 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.384311    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2050bd_da88_4923_b686_4c9e70226b59.slice/crio-0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d WatchSource:0}: Error finding container 0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d: Status 404 returned error can't find the container with id 0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d
Nov 03 03:00:33.385426 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.385407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jb7sk"
Nov 03 03:00:33.386953 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.386932    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c603e8_5a0a_42af_a866_3439e1fdef56.slice/crio-cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d WatchSource:0}: Error finding container cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d: Status 404 returned error can't find the container with id cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d
Nov 03 03:00:33.393875 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.393854    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:00:33.394395 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.394377    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44942777_b4b6_4264_aeff_7c88943cf14e.slice/crio-d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34 WatchSource:0}: Error finding container d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34: Status 404 returned error can't find the container with id d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34
Nov 03 03:00:33.399972 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.399955    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:00:33.403248 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:00:33.403212    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09cc1ed_086f_4b12_a9ff_e465fbe94038.slice/crio-80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d WatchSource:0}: Error finding container 80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d: Status 404 returned error can't find the container with id 80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d
Nov 03 03:00:33.643138 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.643031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:33.643138 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.643126    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:33.643357 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.643243    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:34.643221015 +0000 UTC m=+4.069954102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:33.744381 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.744341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:33.744597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:33.744426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:33.744597 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.744532    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:33.744597 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.744531    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:00:33.744597 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.744584    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:00:33.744597 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.744597    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:33.744828 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.744619    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:34.744599724 +0000 UTC m=+4.171332824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:33.744828 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:33.744642    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:34.744627081 +0000 UTC m=+4.171360185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:34.059822 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.059642    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4" event={"ID":"ff3b7b52-b0ad-4965-ba72-39330944d766","Type":"ContainerStarted","Data":"7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338"}
Nov 03 03:00:34.061101 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.061063    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xflz" event={"ID":"35125e2e-9a2d-4577-98ca-5966cfe1d3a1","Type":"ContainerStarted","Data":"f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f"}
Nov 03 03:00:34.062360 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.062328    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77z26" event={"ID":"81335b4a-183a-49d7-81d0-8bff20d94e77","Type":"ContainerStarted","Data":"a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d"}
Nov 03 03:00:34.063767 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.063745    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jb7sk" event={"ID":"44942777-b4b6-4264-aeff-7c88943cf14e","Type":"ContainerStarted","Data":"d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34"}
Nov 03 03:00:34.065143 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.065108    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d"}
Nov 03 03:00:34.066421 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.066393    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmpwj" event={"ID":"7a2050bd-da88-4923-b686-4c9e70226b59","Type":"ContainerStarted","Data":"0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d"}
Nov 03 03:00:34.067886 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.067853    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4msjq" event={"ID":"64651a3d-f058-40c3-9171-53ba4e35d04a","Type":"ContainerStarted","Data":"61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1"}
Nov 03 03:00:34.069110 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.069090    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgh8q" event={"ID":"794ed6bd-5ce9-4aea-99ab-75607b2cd846","Type":"ContainerStarted","Data":"979a30f47fb0ac699a76e59d660dfd9691e00ddbb9f8e1a1acaebb60bfbd86e7"}
Nov 03 03:00:34.070424 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.070405    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerStarted","Data":"80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d"}
Nov 03 03:00:34.127618 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.127574    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:33 +0000 UTC" deadline="2028-08-18 07:24:11.006334672 +0000 UTC"
Nov 03 03:00:34.127618 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.127610    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15700h23m36.878728917s"
Nov 03 03:00:34.654157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.654114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:34.654443 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.654422    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:34.654522 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.654510    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:36.654486292 +0000 UTC m=+6.081219414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:34.754686 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.754563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:34.754856 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:34.754691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:34.754856 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.754830    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:34.754982 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.754897    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:36.754877327 +0000 UTC m=+6.181610415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:34.755054 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.754992    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:00:34.755054 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.755009    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:00:34.755054 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.755022    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:34.755216 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:34.755058    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:36.755046159 +0000 UTC m=+6.181779247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:35.052280 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:35.052249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:35.052467 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:35.052281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:35.052467 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:35.052358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:35.052636 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:35.052490    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:35.052753 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:35.052649    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:35.052815 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:35.052746    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:35.128494 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:35.128434    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:33 +0000 UTC" deadline="2028-04-28 05:21:31.112655582 +0000 UTC"
Nov 03 03:00:35.128494 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:35.128488    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13010h20m55.984170048s"
Nov 03 03:00:36.086252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:36.086037    2576 generic.go:356] "Generic (PLEG): container finished" podID="f0a743d363639b28a6d7229f5e9636d9" containerID="adf936ee0d17157f6cd6ddd2e537f582de8702665abd5e339e745d472969820d" exitCode=0
Nov 03 03:00:36.086506 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:36.086283    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"f0a743d363639b28a6d7229f5e9636d9","Type":"ContainerDied","Data":"adf936ee0d17157f6cd6ddd2e537f582de8702665abd5e339e745d472969820d"}
Nov 03 03:00:36.675198 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:36.675106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:36.675681 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.675343    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:36.675681 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.675415    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:40.675391242 +0000 UTC m=+10.102124327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:36.775838 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:36.775797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:36.776023 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:36.775858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:36.776023 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.775986    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:00:36.776023 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.776001    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:00:36.776023 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.776011    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:36.776165 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.776070    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:40.776051544 +0000 UTC m=+10.202784650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:36.776165 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.776152    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:36.776254 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:36.776191    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:40.776179443 +0000 UTC m=+10.202912541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:37.053158 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.052743    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:37.053158 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:37.052894    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:37.053158 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.052964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:37.053158 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:37.053043    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:37.053721 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.053550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:37.053721 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:37.053684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:37.093664 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.093609    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"f0a743d363639b28a6d7229f5e9636d9","Type":"ContainerStarted","Data":"ad75cc82740e9ae1ddc6f9c8be0b0b793f0a40ffdb9a0f55b1f9fdd56d6becc2"}
Nov 03 03:00:37.100506 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.100468    2576 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="af83ccf6dd21a71f70106fe25c5a64d6b7699d268cda896b4818ddfc2e502702" exitCode=0
Nov 03 03:00:37.100744 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.100530    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerDied","Data":"af83ccf6dd21a71f70106fe25c5a64d6b7699d268cda896b4818ddfc2e502702"}
Nov 03 03:00:37.364243 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:37.364160    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" podStartSLOduration=4.364138899 podStartE2EDuration="4.364138899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:37.223124071 +0000 UTC m=+6.649857180" watchObservedRunningTime="2026-11-03 03:00:37.364138899 +0000 UTC m=+6.790872010"
Nov 03 03:00:38.109009 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:38.108971    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xflz" event={"ID":"35125e2e-9a2d-4577-98ca-5966cfe1d3a1","Type":"ContainerStarted","Data":"1661f0033c6f9cf8fc86249bb80675c634abcc99777bb90ec4217435043c8f25"}
Nov 03 03:00:38.245516 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:38.244659    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5xflz" podStartSLOduration=2.192446855 podStartE2EDuration="6.244636315s" totalImagesPullingTime="4.05218946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:38.244138642 +0000 UTC m=+7.670871750" watchObservedRunningTime="2026-11-03 03:00:38.244636315 +0000 UTC m=+7.671369426"
Nov 03 03:00:39.051869 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:39.051785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:39.052059 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:39.051950    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:39.052059 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:39.052043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:39.052206 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:39.052136    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:39.052206 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:39.052191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:39.052325 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:39.052261    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:39.115740 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:39.115662    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"8452f8e9cc16a665620f2f318ca698f0","Type":"ContainerStarted","Data":"7cd13ed9e71a55608bbdf2859323b205e4c650e9835f5c515f0ede1d94e06995"}
Nov 03 03:00:40.120233 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.120172    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4" event={"ID":"ff3b7b52-b0ad-4965-ba72-39330944d766","Type":"ContainerStarted","Data":"28551db142c6da9a91e9334ec1bf730a780a12de22af3fdf8f8c9ecb48021d38"}
Nov 03 03:00:40.711975 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.711930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:40.712177 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.712120    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:40.712245 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.712185    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:48.712164009 +0000 UTC m=+18.138897094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:40.812919 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.812854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:40.812919 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.812923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:40.813164 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.813096    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:00:40.813164 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.813112    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:00:40.813164 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.813123    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:40.813323 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.813172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:00:48.81315915 +0000 UTC m=+18.239892237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:40.813323 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.813232    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:40.813323 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:40.813269    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:00:48.813256837 +0000 UTC m=+18.239989920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:40.852478 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.852422    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:40.853689 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.853663    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:40.978552 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:40.978444    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal" podStartSLOduration=7.9784205660000005 podStartE2EDuration="7.978420566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:39.30975215 +0000 UTC m=+8.736485260" watchObservedRunningTime="2026-11-03 03:00:40.978420566 +0000 UTC m=+10.405153673"
Nov 03 03:00:41.056142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:41.055643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:41.056142 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:41.055767    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:41.056142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:41.055853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:41.056142 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:41.055942    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:41.056142 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:41.055996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:41.056142 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:41.056077    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:41.127783 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:41.127705    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jb7sk" event={"ID":"44942777-b4b6-4264-aeff-7c88943cf14e","Type":"ContainerStarted","Data":"ba986d3f84cc2ed9819164c8bb22ab5e7a5b61927c508b6e5472ad792469c6c3"}
Nov 03 03:00:42.134014 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.133765    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4msjq" event={"ID":"64651a3d-f058-40c3-9171-53ba4e35d04a","Type":"ContainerStarted","Data":"9c7132d600e7633b2cac801e6e20a6710e00cfcdfcb244805eb57ebe0f7ff00d"}
Nov 03 03:00:42.138352 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.138326    2576 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="792a9767c89ad81d2cad39797d861c1d062ece44a38187455fc276e0da72be6d" exitCode=0
Nov 03 03:00:42.138488 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.138421    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerDied","Data":"792a9767c89ad81d2cad39797d861c1d062ece44a38187455fc276e0da72be6d"}
Nov 03 03:00:42.140220 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.140146    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77z26" event={"ID":"81335b4a-183a-49d7-81d0-8bff20d94e77","Type":"ContainerStarted","Data":"aaca46d203cca9810938360c93ac5934f2320c4248cb457b927e78e3fa8a8944"}
Nov 03 03:00:42.232863 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.232834    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 03:00:42.258168 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.258121    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4msjq" podStartSLOduration=1.809887099 podStartE2EDuration="10.258104991s" totalImagesPullingTime="8.448217892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:42.257931675 +0000 UTC m=+11.684664782" watchObservedRunningTime="2026-11-03 03:00:42.258104991 +0000 UTC m=+11.684838099"
Nov 03 03:00:42.258835 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:42.258788    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jb7sk" podStartSLOduration=2.670365572 podStartE2EDuration="10.258625712s" totalImagesPullingTime="7.58826014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:41.259198978 +0000 UTC m=+10.685932086" watchObservedRunningTime="2026-11-03 03:00:42.258625712 +0000 UTC m=+11.685358818"
Nov 03 03:00:43.010971 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.010836    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T03:00:42.232857812Z","UUID":"4ddd6d8f-a589-48b9-b1d9-3c032e206c85","Handler":null,"Name":"","Endpoint":""}
Nov 03 03:00:43.012803 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.012779    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 03:00:43.012961 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.012811    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 03:00:43.051468 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.051434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:43.051468 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.051465    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:43.051468 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.051472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:43.051720 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:43.051583    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:43.051720 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:43.051640    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:43.051720 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:43.051695    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:43.144288 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:43.144198    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4" event={"ID":"ff3b7b52-b0ad-4965-ba72-39330944d766","Type":"ContainerStarted","Data":"f1995042d0c97c2ce68d6d970c1c21266fdf0fd8a3553c32b4d954732930935a"}
Nov 03 03:00:44.147819 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:44.147377    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmpwj" event={"ID":"7a2050bd-da88-4923-b686-4c9e70226b59","Type":"ContainerStarted","Data":"167404b31b9c04de25eba0ccedd0c9134da814e1cb3ef247f524a865c5a6112f"}
Nov 03 03:00:44.149837 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:44.149810    2576 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="f13fd4fdd566821d0db7dc4c59b36688eee4181a26d93ae9ec361bebcba28026" exitCode=0
Nov 03 03:00:44.149936 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:44.149850    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerDied","Data":"f13fd4fdd566821d0db7dc4c59b36688eee4181a26d93ae9ec361bebcba28026"}
Nov 03 03:00:44.270615 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:44.270564    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-77z26" podStartSLOduration=4.300587162 podStartE2EDuration="12.270534237s" totalImagesPullingTime="7.969947075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:42.518727054 +0000 UTC m=+11.945460161" watchObservedRunningTime="2026-11-03 03:00:44.270534237 +0000 UTC m=+13.697267339"
Nov 03 03:00:44.270770 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:44.270755    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmpwj" podStartSLOduration=3.406188752 podStartE2EDuration="12.270748189s" totalImagesPullingTime="8.864559437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:44.270492594 +0000 UTC m=+13.697225697" watchObservedRunningTime="2026-11-03 03:00:44.270748189 +0000 UTC m=+13.697481292"
Nov 03 03:00:45.051459 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:45.051422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:45.051459 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:45.051442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:45.051704 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:45.051565    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:45.051704 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:45.051578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:45.051704 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:45.051685    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:45.051856 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:45.051747    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:45.154796 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:45.154721    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4" event={"ID":"ff3b7b52-b0ad-4965-ba72-39330944d766","Type":"ContainerStarted","Data":"2b901620074cd6ad2bebd960e6fdf38b17eb04514a4ce71395bef4a9da3afa09"}
Nov 03 03:00:45.273242 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:45.273191    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4" podStartSLOduration=2.872660054 podStartE2EDuration="13.273177614s" totalImagesPullingTime="10.40051756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:45.272726159 +0000 UTC m=+14.699459254" watchObservedRunningTime="2026-11-03 03:00:45.273177614 +0000 UTC m=+14.699910717"
Nov 03 03:00:46.160062 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:46.160025    2576 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="4e1a2b31125cda4b7252dc3e761c22698d3fa74d310ddbf22e0af0c53289ea4d" exitCode=0
Nov 03 03:00:46.160702 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:46.160146    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerDied","Data":"4e1a2b31125cda4b7252dc3e761c22698d3fa74d310ddbf22e0af0c53289ea4d"}
Nov 03 03:00:47.052157 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:47.052117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:47.052340 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:47.052130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:47.052340 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:47.052267    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:47.052340 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:47.052295    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:47.052491 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:47.052380    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:47.052491 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:47.052452    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:48.771832 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:48.771513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:48.772441 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.771676    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:48.772441 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.771944    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:04.771922202 +0000 UTC m=+34.198655282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:00:48.873029 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:48.872990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:48.873216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:48.873051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:48.873469 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.873441    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:00:48.873469 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.873470    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:00:48.873647 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.873481    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:48.873647 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.873565    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:04.873530222 +0000 UTC m=+34.300263319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:00:48.873647 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.873543    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:48.873647 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:48.873631    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:04.873614496 +0000 UTC m=+34.300347592 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:00:49.051744 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:49.051637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:49.051880 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:49.051776    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:49.052171 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:49.052153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:49.052275 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:49.052254    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:49.052346 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:49.052328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:49.052439 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:49.052421    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:50.008052 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:50.008017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:50.170233 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:50.170201    2576 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="b8aff79b4aef145e5e46719b51199f54ce10f19f29131685ec5eee4070d7ce79" exitCode=0
Nov 03 03:00:50.170383 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:50.170248    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerDied","Data":"b8aff79b4aef145e5e46719b51199f54ce10f19f29131685ec5eee4070d7ce79"}
Nov 03 03:00:50.264362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:50.264292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:51.052955 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:51.052924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:51.053576 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:51.052989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:51.053686 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:51.053601    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:51.053686 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:51.053029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:51.053686 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:51.053659    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:51.053837 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:51.053738    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:51.176299 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:51.176266    2576 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="11ac97b2ce165a6b489f2e051f946ff7c72ff2d384419706ed9cc34b4229af1e" exitCode=0
Nov 03 03:00:51.176470 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:51.176330    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerDied","Data":"11ac97b2ce165a6b489f2e051f946ff7c72ff2d384419706ed9cc34b4229af1e"}
Nov 03 03:00:52.183180 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:52.183140    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" event={"ID":"f09cc1ed-086f-4b12-a9ff-e465fbe94038","Type":"ContainerStarted","Data":"9fd8d8c5a1cb82884a2bc89d9950713a6d4df30e1d38ed5bf6dde71c8ce0c720"}
Nov 03 03:00:53.051565 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:53.051515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:53.051746 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:53.051526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:53.051746 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:53.051665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:53.051746 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:53.051667    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:53.051909 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:53.051776    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:53.051909 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:53.051871    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:53.172417 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:53.172363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:53.365870 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:53.365836    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:54.903358 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:54.903316    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:54.903912 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:54.903479    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 03:00:54.903986 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:54.903952    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:00:55.019614 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.019535    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rb9rv" podStartSLOduration=9.39363566 podStartE2EDuration="23.019516383s" totalImagesPullingTime="13.625880723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:52.406207798 +0000 UTC m=+21.832940902" watchObservedRunningTime="2026-11-03 03:00:55.019516383 +0000 UTC m=+24.446249489"
Nov 03 03:00:55.052285 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.052256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:55.052285 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.052293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:55.052569 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:55.052391    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:55.052569 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.052453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:55.052682 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:55.052584    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:55.052682 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:55.052632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:55.092240 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.092197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:55.190950 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.190857    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgh8q" event={"ID":"794ed6bd-5ce9-4aea-99ab-75607b2cd846","Type":"ContainerStarted","Data":"6c1e8b268d6c6a4eb36e30832097c13969f9ca6fab5d160daa6b9465123ed744"}
Nov 03 03:00:55.312207 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.312149    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hgh8q" podStartSLOduration=2.347597189 podStartE2EDuration="23.31213465s" totalImagesPullingTime="20.964537461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:55.312069901 +0000 UTC m=+24.738803005" watchObservedRunningTime="2026-11-03 03:00:55.31213465 +0000 UTC m=+24.738867752"
Nov 03 03:00:55.563478 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.563370    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:55.849743 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:55.849650    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:56.046885 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:56.046848    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:56.216928 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:56.216884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:57.051620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:57.051585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:57.051620 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:57.051618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:57.052082 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:57.051591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:57.052082 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:57.051739    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:57.052082 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:57.051869    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:57.052082 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:57.051941    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:57.528559 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:57.528506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:58.363124 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:58.363099    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77z26_81335b4a-183a-49d7-81d0-8bff20d94e77/dns-node-resolver/0.log"
Nov 03 03:00:59.017858 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.017783    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:59.051766 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.051728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:00:59.051935 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.051728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:00:59.051935 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:59.051844    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:00:59.052060 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.051941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:00:59.052060 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:59.051945    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:00:59.052060 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:00:59.052026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:00:59.202244 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202209    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:00:59.202578 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202533    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3c603e8-5a0a-42af-a866-3439e1fdef56" containerID="c1a03c31d13ef4cff44a73b7727cc26c4b9279d71688f898346b09c164815a00" exitCode=1
Nov 03 03:00:59.202659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202590    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"ba2811185377dffc790dc3b50d2e39e27e756f6d9112f519e30144643dca8551"}
Nov 03 03:00:59.202721 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202677    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"d86b59f5bfc033ff06df81d9286f8c9891e59c93f483b87a88357f212f82aa69"}
Nov 03 03:00:59.202775 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202729    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"e2827b44424d905fe8dba3198d4007154f9e9d48febf9d262b36dd8accc1fb76"}
Nov 03 03:00:59.202830 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202779    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"3b24dfce810c7b8406d71793c2dad9a10ab12da2fea061b6e1cf13557e5eded6"}
Nov 03 03:00:59.202830 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202824    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerDied","Data":"c1a03c31d13ef4cff44a73b7727cc26c4b9279d71688f898346b09c164815a00"}
Nov 03 03:00:59.202928 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.202874    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"567564d00ea20358cddff4f3a30fd18f2fe079250355dd5e08cf42c748d2811a"}
Nov 03 03:00:59.285174 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.285096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:00:59.363804 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.363776    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4msjq_64651a3d-f058-40c3-9171-53ba4e35d04a/node-ca/0.log"
Nov 03 03:00:59.841750 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:00:59.841712    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:00.029610 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:00.029577    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:00.213757 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:00.213726    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:01.035762 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.035731    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:01.052409 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.052382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:01.052572 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.052514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:01.052572 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:01.052540    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:01.052669 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.052590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:01.052721 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:01.052659    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:01.052773 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:01.052746    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:01.201112 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.201030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:01.210403 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.210381    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:01:01.210776 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.210754    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"19f1d1a78dc1ad727b676bd05106f951f0aad35a918c2bbd38111f759464ea62"}
Nov 03 03:01:01.386580 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:01.386533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:02.297174 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:02.297144    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:03.051642 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.051597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:03.051832 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.051669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:03.051832 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.051687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:03.051832 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:03.051798    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:03.052008 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:03.051849    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:03.052008 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:03.051928    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:03.218183 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.217940    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:01:03.218443 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.218423    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"78e7d879311069cffa709e6d2f74c17c3ac67e62b4973339b1e33066130d45c9"}
Nov 03 03:01:03.218690 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.218672    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:03.218854 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.218838    2576 scope.go:122] "RemoveContainer" containerID="c1a03c31d13ef4cff44a73b7727cc26c4b9279d71688f898346b09c164815a00"
Nov 03 03:01:03.235241 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.235219    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:03.353414 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.353378    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:03.554958 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.554869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:03.743224 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.743191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:03.906875 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:03.906835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:04.101919 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.101885    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:04.224393 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.224322    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:01:04.224746 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.224721    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" event={"ID":"a3c603e8-5a0a-42af-a866-3439e1fdef56","Type":"ContainerStarted","Data":"eb6c3a57e0cc004993b8db3355ff1ba0ddcc0e4b25ca146d42d1989624ff6cbd"}
Nov 03 03:01:04.224991 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.224966    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:04.225119 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.225036    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:04.240492 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.240470    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:04.272762 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.272739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:04.374792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.374747    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5" podStartSLOduration=7.38931295 podStartE2EDuration="32.374733643s" totalImagesPullingTime="24.985420693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:04.374579584 +0000 UTC m=+33.801312689" watchObservedRunningTime="2026-11-03 03:01:04.374733643 +0000 UTC m=+33.801466738"
Nov 03 03:01:04.789541 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.789495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:04.789801 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.789653    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:01:04.789801 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.789717    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs podName:996d2ead-9513-41c2-acd6-1ced1f624131 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:36.789701044 +0000 UTC m=+66.216434142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs") pod "network-metrics-daemon-6k7g2" (UID: "996d2ead-9513-41c2-acd6-1ced1f624131") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 03:01:04.890816 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.890784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:04.890949 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.890824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:04.890949 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.890931    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 03:01:04.890949 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.890937    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 03:01:04.891041 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.890957    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 03:01:04.891041 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.890967    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bnz9k for pod openshift-network-diagnostics/network-check-target-568md: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:01:04.891041 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.890998    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret podName:5000ce1d-93ee-4bed-bf15-7ecb2d1301e9 nodeName:}" failed. No retries permitted until 2026-11-03 03:01:36.890981736 +0000 UTC m=+66.317714831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret") pod "global-pull-secret-syncer-rqm5c" (UID: "5000ce1d-93ee-4bed-bf15-7ecb2d1301e9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 03:01:04.891041 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:04.891013    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k podName:63321407-0b4f-4f4c-9f17-19aaf6ef081a nodeName:}" failed. No retries permitted until 2026-11-03 03:01:36.891007169 +0000 UTC m=+66.317740250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bnz9k" (UniqueName: "kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k") pod "network-check-target-568md" (UID: "63321407-0b4f-4f4c-9f17-19aaf6ef081a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 03:01:04.915485 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:04.915454    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:05.051342 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:05.051269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:05.051342 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:05.051286    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:05.051535 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:05.051270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:05.051535 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:05.051377    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:05.051535 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:05.051465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:05.051654 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:05.051541    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:05.102304 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:05.102266    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:05.431282 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:05.431247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:05.865892 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:05.865858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:06.077156 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:06.077123    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:06.275443 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:06.275362    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:07.051912 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:07.051874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:07.051912 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:07.051889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:07.052453 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:07.052100    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:07.052453 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:07.052127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:07.052453 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:07.052209    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:07.052453 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:07.052275    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:07.434079 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:07.434003    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:07.637720 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:07.637685    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:07.841194 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:07.841113    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:08.041657 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:08.041626    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:08.222817 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:08.222785    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:08.631871 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:08.631837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:08.792359 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:08.792328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:09.051756 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:09.051660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:09.051971 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:09.051665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:09.051971 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:09.051791    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:09.051971 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:09.051665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:09.051971 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:09.051850    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:09.051971 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:09.051917    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:09.545896 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:09.545854    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:09.727280 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:09.727231    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:09.896299 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:09.896264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:10.051054 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:10.051025    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:10.249610 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:10.249492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:11.052296 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.052249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:11.052812 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:11.052354    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:11.052812 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.052368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:11.052812 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.052391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:11.052812 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:11.052479    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:11.052812 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:11.052562    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:11.077652 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.077624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:11.468672 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.468632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:11.655478 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.655446    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:11.982858 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:11.982815    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:12.174129 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:12.174093    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:12.363269 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:12.363238    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:12.854884 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:12.854790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:13.051387 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:13.051343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:13.051601 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:13.051343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:13.051601 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:13.051450    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:13.051601 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:13.051356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:13.051601 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:13.051527    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:13.051824 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:13.051631    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:13.255928 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:13.255897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:13.442374 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:13.442343    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:13.657021 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:13.656990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:14.029134 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:14.029055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:14.221648 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:14.221587    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:14.402499 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:14.402467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:15.051811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:15.051772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:15.051811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:15.051799    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:15.052067 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:15.051778    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:15.052067 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:15.051910    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:15.052067 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:15.052012    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:15.052226 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:15.052072    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:17.051465 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:17.051428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:17.051929 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:17.051432    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:17.051929 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:17.051536    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:17.051929 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:17.051442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:17.051929 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:17.051620    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:17.051929 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:17.051709    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:18.503362 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:18.503332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:18.700742 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:18.700703    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:18.863901 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:18.863877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:19.051269 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.051242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:19.051454 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.051279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:19.051454 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:19.051368    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:19.051454 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:19.051429    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:19.051593 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.051480    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:19.051627 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:19.051602    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:19.079367 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.079333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:19.412196 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.412165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:19.585749 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.585715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:19.769064 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.768982    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:19.959437 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:19.959406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:21.052942 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:21.052907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:21.053351 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:21.053000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:21.053351 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:21.053037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:21.053351 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:21.053078    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:21.053351 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:21.053099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:21.053351 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:21.053145    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:23.051888 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.051853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:23.051888 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.051849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:23.052402 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:23.052060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:23.052402 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.052079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:23.052402 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:23.052138    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:23.052402 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:23.052175    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:23.191369 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.191333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:23.372036 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.371993    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:23.730737 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.730671    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:23.924953 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:23.924921    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:24.720037 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:24.719779    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rqm5c"]
Nov 03 03:01:24.720438 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:24.720140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:24.720438 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:24.720259    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:24.724608 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:24.724586    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-568md"]
Nov 03 03:01:24.724780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:24.724697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:24.724846 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:24.724803    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:24.725367 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:24.725339    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6k7g2"]
Nov 03 03:01:24.725494 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:24.725440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:24.725593 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:24.725568    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:26.052234 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:26.052153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:26.052628 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:26.052263    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:26.052628 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:26.052319    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:26.052628 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:26.052410    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:27.051870 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:27.051830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:27.052034 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:27.051975    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:27.276349 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:27.276277    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:01:27.276349 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:27.276319    2576 generic.go:356] "Generic (PLEG): container finished" podID="44942777-b4b6-4264-aeff-7c88943cf14e" containerID="ba986d3f84cc2ed9819164c8bb22ab5e7a5b61927c508b6e5472ad792469c6c3" exitCode=1
Nov 03 03:01:27.276954 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:27.276382    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jb7sk" event={"ID":"44942777-b4b6-4264-aeff-7c88943cf14e","Type":"ContainerDied","Data":"ba986d3f84cc2ed9819164c8bb22ab5e7a5b61927c508b6e5472ad792469c6c3"}
Nov 03 03:01:27.276954 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:27.276768    2576 scope.go:122] "RemoveContainer" containerID="ba986d3f84cc2ed9819164c8bb22ab5e7a5b61927c508b6e5472ad792469c6c3"
Nov 03 03:01:28.052249 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.052214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:28.052407 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:28.052319    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-568md" podUID="63321407-0b4f-4f4c-9f17-19aaf6ef081a"
Nov 03 03:01:28.052407 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.052378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:28.052526 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:28.052490    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqm5c" podUID="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9"
Nov 03 03:01:28.172701 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.172665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:28.280015 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.279990    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:01:28.280645 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.280047    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jb7sk" event={"ID":"44942777-b4b6-4264-aeff-7c88943cf14e","Type":"ContainerStarted","Data":"db843c8b18f0ed699becdf8c574bae80e28b8f8b96ca3c931a3e807eabaaf6a9"}
Nov 03 03:01:28.384219 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.384185    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:28.600872 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.600838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:28.824811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:28.824778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:29.051602 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.051506    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:29.051727 ip-10-0-138-11 kubenswrapper[2576]: E1103 03:01:29.051648    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6k7g2" podUID="996d2ead-9513-41c2-acd6-1ced1f624131"
Nov 03 03:01:29.315925 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.315851    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeReady"
Nov 03 03:01:29.316291 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.316000    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 03:01:29.484810 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.484779    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hgj4z"]
Nov 03 03:01:29.501080 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.501043    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:29.505382 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.505361    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8r26d"]
Nov 03 03:01:29.505498 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.505483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.521450 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.521425    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.548050 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.548022    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mrlzv"]
Nov 03 03:01:29.558109 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc29a623-58c0-41a7-9167-926cc8e5b8c8-tmp-dir\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.558109 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.558324 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc29a623-58c0-41a7-9167-926cc8e5b8c8-config-volume\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.558324 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d44eda-e0e2-4bea-afeb-160eebdc1710-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.558422 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc29a623-58c0-41a7-9167-926cc8e5b8c8-metrics-tls\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.558422 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwr2\" (UniqueName: \"kubernetes.io/projected/bc29a623-58c0-41a7-9167-926cc8e5b8c8-kube-api-access-tfwr2\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.558512 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d44eda-e0e2-4bea-afeb-160eebdc1710-data-volume\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.558512 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d44eda-e0e2-4bea-afeb-160eebdc1710-crio-socket\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.558621 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.558504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsp96\" (UniqueName: \"kubernetes.io/projected/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-api-access-dsp96\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.559239 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.559226    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgj4z"]
Nov 03 03:01:29.559323 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.559313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.572068 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.572010    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 03:01:29.572284 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.572269    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 03:01:29.572349 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.572306    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 03:01:29.603442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.603415    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snr2z\""
Nov 03 03:01:29.603442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.603424    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 03:01:29.603637 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.603414    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz9pb\""
Nov 03 03:01:29.604169 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.604152    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8r26d"]
Nov 03 03:01:29.642905 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.642870    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrlzv"]
Nov 03 03:01:29.659643 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d44eda-e0e2-4bea-afeb-160eebdc1710-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.659807 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc29a623-58c0-41a7-9167-926cc8e5b8c8-metrics-tls\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.659807 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwr2\" (UniqueName: \"kubernetes.io/projected/bc29a623-58c0-41a7-9167-926cc8e5b8c8-kube-api-access-tfwr2\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.659807 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f023c708-c71a-4dcb-99cd-f604cb0088ce-cert\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.659807 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d44eda-e0e2-4bea-afeb-160eebdc1710-data-volume\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.659807 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d44eda-e0e2-4bea-afeb-160eebdc1710-crio-socket\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.659807 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsp96\" (UniqueName: \"kubernetes.io/projected/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-api-access-dsp96\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.660080 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d44eda-e0e2-4bea-afeb-160eebdc1710-crio-socket\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.660080 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc29a623-58c0-41a7-9167-926cc8e5b8c8-tmp-dir\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.660080 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.660080 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.659988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptbp\" (UniqueName: \"kubernetes.io/projected/f023c708-c71a-4dcb-99cd-f604cb0088ce-kube-api-access-2ptbp\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.660080 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.660019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc29a623-58c0-41a7-9167-926cc8e5b8c8-config-volume\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.660250 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.660106    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d44eda-e0e2-4bea-afeb-160eebdc1710-data-volume\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.660293 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.660274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc29a623-58c0-41a7-9167-926cc8e5b8c8-tmp-dir\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.660539 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.660490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.663904 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.663884    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc29a623-58c0-41a7-9167-926cc8e5b8c8-metrics-tls\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.671095 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.671075    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 03:01:29.671204 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.671164    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 03:01:29.671333 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.671320    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 03:01:29.678730 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.678713    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pw7w9\""
Nov 03 03:01:29.678821 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.678752    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 03:01:29.678821 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.678767    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 03:01:29.680635 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.680618    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc29a623-58c0-41a7-9167-926cc8e5b8c8-config-volume\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.682035 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.682015    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d44eda-e0e2-4bea-afeb-160eebdc1710-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.720831 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.720792    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwr2\" (UniqueName: \"kubernetes.io/projected/bc29a623-58c0-41a7-9167-926cc8e5b8c8-kube-api-access-tfwr2\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.722900 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.722870    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsp96\" (UniqueName: \"kubernetes.io/projected/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-api-access-dsp96\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.760597 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.760544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f023c708-c71a-4dcb-99cd-f604cb0088ce-cert\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.760760 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.760650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptbp\" (UniqueName: \"kubernetes.io/projected/f023c708-c71a-4dcb-99cd-f604cb0088ce-kube-api-access-2ptbp\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.763198 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.763174    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f023c708-c71a-4dcb-99cd-f604cb0088ce-cert\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.823440 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.823375    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptbp\" (UniqueName: \"kubernetes.io/projected/f023c708-c71a-4dcb-99cd-f604cb0088ce-kube-api-access-2ptbp\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.832740 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.832710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:29.856383 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.856345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:29.882115 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.882084    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:29.898854 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.898811    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:29.999461 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:29.999256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:30.051928 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.051892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:30.052129 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.051938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:30.108935 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.108908    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 03:01:30.108935 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.108931    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 03:01:30.109113 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.108973    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\""
Nov 03 03:01:30.109113 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.109036    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 03:01:30.109113 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.109053    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\""
Nov 03 03:01:30.196405 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.196373    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-dfdd8856b-dgsz5"]
Nov 03 03:01:30.209102 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.209079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.260470 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.260433    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfdd8856b-dgsz5"]
Nov 03 03:01:30.264922 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.264892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.265072 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.264931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.265072 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.264975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.265072 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.265050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.265072 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.265064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.265248 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.265087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.265248 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.265140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.266561 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.266517    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 03:01:30.266561 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.266517    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v5wbd\""
Nov 03 03:01:30.266718 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.266671    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 03:01:30.267464 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.267446    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 03:01:30.267612 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.267495    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 03:01:30.267727 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.267705    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 03:01:30.267762 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.267740    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 03:01:30.267793 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.267781    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 03:01:30.311991 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.311896    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8r26d"]
Nov 03 03:01:30.314934 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:30.314911    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc29a623_58c0_41a7_9167_926cc8e5b8c8.slice/crio-2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3 WatchSource:0}: Error finding container 2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3: Status 404 returned error can't find the container with id 2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3
Nov 03 03:01:30.363283 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.363213    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrlzv"]
Nov 03 03:01:30.366123 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366244 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366244 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366244 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366244 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.366442 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:30.366322    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf023c708_c71a_4dcb_99cd_f604cb0088ce.slice/crio-284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0 WatchSource:0}: Error finding container 284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0: Status 404 returned error can't find the container with id 284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0
Nov 03 03:01:30.366763 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366735    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.367022 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.366999    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.367085 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.367008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.367120 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.367105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.368850 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.368829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.368959 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.368944    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.372989 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.372966    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hgj4z"]
Nov 03 03:01:30.375998 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:30.375971    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d44eda_e0e2_4bea_afeb_160eebdc1710.slice/crio-593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b WatchSource:0}: Error finding container 593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b: Status 404 returned error can't find the container with id 593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b
Nov 03 03:01:30.427412 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.427374    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.527854 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.527816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:30.927678 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:30.927646    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfdd8856b-dgsz5"]
Nov 03 03:01:30.931063 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:30.931033    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f28420_10c0_44c8_a199_296d755fa0dc.slice/crio-ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f WatchSource:0}: Error finding container ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f: Status 404 returned error can't find the container with id ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f
Nov 03 03:01:31.052707 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.052664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:31.105942 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.105907    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\""
Nov 03 03:01:31.106114 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.105979    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 03:01:31.287451 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.287366    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8r26d" event={"ID":"bc29a623-58c0-41a7-9167-926cc8e5b8c8","Type":"ContainerStarted","Data":"2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3"}
Nov 03 03:01:31.288741 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.288718    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgj4z" event={"ID":"c7d44eda-e0e2-4bea-afeb-160eebdc1710","Type":"ContainerStarted","Data":"02e9d01e0471df682251662c1d299c27d9bd425cd7c45309be51ac30dc4c200a"}
Nov 03 03:01:31.288830 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.288764    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgj4z" event={"ID":"c7d44eda-e0e2-4bea-afeb-160eebdc1710","Type":"ContainerStarted","Data":"593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b"}
Nov 03 03:01:31.289750 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.289734    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfdd8856b-dgsz5" event={"ID":"05f28420-10c0-44c8-a199-296d755fa0dc","Type":"ContainerStarted","Data":"ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f"}
Nov 03 03:01:31.290762 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:31.290744    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrlzv" event={"ID":"f023c708-c71a-4dcb-99cd-f604cb0088ce","Type":"ContainerStarted","Data":"284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0"}
Nov 03 03:01:33.298147 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:33.298104    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgj4z" event={"ID":"c7d44eda-e0e2-4bea-afeb-160eebdc1710","Type":"ContainerStarted","Data":"17bd328d63315ef4c63d0efbca87ef6dcb38bcedce684f31df0d27ce7f05b690"}
Nov 03 03:01:34.079391 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:34.079355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:34.282187 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:34.282149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:35.307683 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:35.307641    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8r26d" event={"ID":"bc29a623-58c0-41a7-9167-926cc8e5b8c8","Type":"ContainerStarted","Data":"93a3521b0a3fe3490443508cc95a00d0704ce2ad7d8fb9026e3771647c5a80a6"}
Nov 03 03:01:35.308509 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:35.307697    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8r26d" event={"ID":"bc29a623-58c0-41a7-9167-926cc8e5b8c8","Type":"ContainerStarted","Data":"f75c11f4f8572dde8178cf49993d0486d1f9cae27f2bfcf44921c019349cbfd3"}
Nov 03 03:01:35.308509 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:35.307833    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:35.332023 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:35.331984    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:35.452046 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:35.451446    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8r26d" podStartSLOduration=1.8741226229999999 podStartE2EDuration="6.451431541s" totalImagesPullingTime="4.577308918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:35.450506768 +0000 UTC m=+64.877239874" watchObservedRunningTime="2026-11-03 03:01:35.451431541 +0000 UTC m=+64.878164695"
Nov 03 03:01:35.584257 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:35.584224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:36.091190 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.091154    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:36.240666 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.240639    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:36.315296 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.315214    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrlzv" event={"ID":"f023c708-c71a-4dcb-99cd-f604cb0088ce","Type":"ContainerStarted","Data":"beeb6f70789c7596cc64dbadd5550530a276bc9269b57f435ccbbe78d1fd886f"}
Nov 03 03:01:36.318566 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.318522    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hgj4z" event={"ID":"c7d44eda-e0e2-4bea-afeb-160eebdc1710","Type":"ContainerStarted","Data":"84770e9c6d4c2d7b7480cdbe30b9b3476221c2613fa8d66e8f6278a38cabb0fa"}
Nov 03 03:01:36.438453 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.438419    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:36.652537 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.652474    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hgj4z" podStartSLOduration=2.119673788 podStartE2EDuration="7.652454741s" totalImagesPullingTime="5.532780953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:01:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:36.652232827 +0000 UTC m=+66.078965931" watchObservedRunningTime="2026-11-03 03:01:36.652454741 +0000 UTC m=+66.079187848"
Nov 03 03:01:36.774519 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.774484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:36.822429 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.822395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:36.825195 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.825168    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:36.923510 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.923423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:36.923510 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.923489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:36.975050 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.975004    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 03:01:36.975050 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.975030    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 03:01:36.987030 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:36.986995    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:37.035284 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.035256    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 03:01:37.046951 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.046927    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:37.050596 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.050516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:37.092374 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.092348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:37.246937 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.246853    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:37.319500 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.319456    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\""
Nov 03 03:01:37.324881 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.324858    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\""
Nov 03 03:01:37.325380 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.325353    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfdd8856b-dgsz5" event={"ID":"05f28420-10c0-44c8-a199-296d755fa0dc","Type":"ContainerStarted","Data":"dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1"}
Nov 03 03:01:37.327846 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.327828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:37.334043 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.334021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:37.450092 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.450034    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-dfdd8856b-dgsz5" podStartSLOduration=1.322226111 podStartE2EDuration="7.450017818s" totalImagesPullingTime="6.127791707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:37.448980434 +0000 UTC m=+66.875713538" watchObservedRunningTime="2026-11-03 03:01:37.450017818 +0000 UTC m=+66.876750921"
Nov 03 03:01:37.450423 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.450402    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mrlzv" podStartSLOduration=2.773931958 podStartE2EDuration="8.450397321s" totalImagesPullingTime="5.676465363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:36.818242417 +0000 UTC m=+66.244975519" watchObservedRunningTime="2026-11-03 03:01:37.450397321 +0000 UTC m=+66.877130424"
Nov 03 03:01:37.464707 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.464672    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"]
Nov 03 03:01:37.467557 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.467528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.481470 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.481442    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6k7g2"]
Nov 03 03:01:37.486371 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.486346    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:37.515131 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:37.515102    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996d2ead_9513_41c2_acd6_1ced1f624131.slice/crio-04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a WatchSource:0}: Error finding container 04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a: Status 404 returned error can't find the container with id 04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a
Nov 03 03:01:37.525216 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.525174    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 03:01:37.525342 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.525322    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 03:01:37.525442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.525429    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8skg7\""
Nov 03 03:01:37.525507 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.525432    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 03:01:37.525507 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.525470    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 03:01:37.525507 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.525481    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 03:01:37.527898 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.527881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.527982 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.527908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.527982 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.527936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150b0a73-6cfc-438d-853f-3302f2232ec2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.527982 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.527962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcn5\" (UniqueName: \"kubernetes.io/projected/150b0a73-6cfc-438d-853f-3302f2232ec2-kube-api-access-6hcn5\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.529689 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.529671    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"]
Nov 03 03:01:37.599613 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.599584    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wwq7p"]
Nov 03 03:01:37.602008 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.601986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629189 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-textfile\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629189 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629401 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-metrics-client-ca\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629401 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.629471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-sys\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.629471 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-root\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629621 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-wtmp\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629621 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-tls\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629750 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.629796 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150b0a73-6cfc-438d-853f-3302f2232ec2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.629845 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcn5\" (UniqueName: \"kubernetes.io/projected/150b0a73-6cfc-438d-853f-3302f2232ec2-kube-api-access-6hcn5\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.629882 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.629869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpls9\" (UniqueName: \"kubernetes.io/projected/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-kube-api-access-rpls9\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.630780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.630754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150b0a73-6cfc-438d-853f-3302f2232ec2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.632047 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.632017    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.632161 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.632080    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.664780 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.664742    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 03:01:37.665231 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.665214    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 03:01:37.665792 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.665776    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ccc7x\""
Nov 03 03:01:37.667173 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.667160    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 03:01:37.705441 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.705400    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcn5\" (UniqueName: \"kubernetes.io/projected/150b0a73-6cfc-438d-853f-3302f2232ec2-kube-api-access-6hcn5\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.707186 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.707160    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rqm5c"]
Nov 03 03:01:37.710943 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:37.710916    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5000ce1d_93ee_4bed_bf15_7ecb2d1301e9.slice/crio-bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a WatchSource:0}: Error finding container bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a: Status 404 returned error can't find the container with id bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a
Nov 03 03:01:37.725539 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.725515    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-568md"]
Nov 03 03:01:37.729430 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:37.729405    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63321407_0b4f_4f4c_9f17_19aaf6ef081a.slice/crio-acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b WatchSource:0}: Error finding container acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b: Status 404 returned error can't find the container with id acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b
Nov 03 03:01:37.730417 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-root\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730495 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-wtmp\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730495 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-tls\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730495 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730688 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-root\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730688 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpls9\" (UniqueName: \"kubernetes.io/projected/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-kube-api-access-rpls9\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730688 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-wtmp\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730817 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-textfile\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730817 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.730817 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.730767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-metrics-client-ca\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.731189 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.731030    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-textfile\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.731189 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.731132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-sys\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.731189 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.731140    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.731646 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.731203    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-sys\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.731646 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.731437    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-metrics-client-ca\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.733052 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.733032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-tls\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.733163 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.733146    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.758104 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.758069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:37.780005 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.779921    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:37.813930 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.813901    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpls9\" (UniqueName: \"kubernetes.io/projected/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-kube-api-access-rpls9\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.913363 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:37.913333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:37.923091 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:37.923064    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e897c78_ce55_4d3f_bcfd_148c8faf01f2.slice/crio-2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f WatchSource:0}: Error finding container 2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f: Status 404 returned error can't find the container with id 2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f
Nov 03 03:01:38.027959 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.027906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:38.158802 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.158773    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"]
Nov 03 03:01:38.161922 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:38.161897    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150b0a73_6cfc_438d_853f_3302f2232ec2.slice/crio-85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d WatchSource:0}: Error finding container 85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d: Status 404 returned error can't find the container with id 85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d
Nov 03 03:01:38.258564 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.258513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:38.329496 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.329453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6k7g2" event={"ID":"996d2ead-9513-41c2-acd6-1ced1f624131","Type":"ContainerStarted","Data":"04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a"}
Nov 03 03:01:38.330736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.330710    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwq7p" event={"ID":"9e897c78-ce55-4d3f-bcfd-148c8faf01f2","Type":"ContainerStarted","Data":"2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f"}
Nov 03 03:01:38.332462 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.332438    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb" event={"ID":"150b0a73-6cfc-438d-853f-3302f2232ec2","Type":"ContainerStarted","Data":"0cfd20bb830b99c67ee08f68851cf9e574f69dbda79532b2710c6e3ca38926f6"}
Nov 03 03:01:38.332617 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.332488    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb" event={"ID":"150b0a73-6cfc-438d-853f-3302f2232ec2","Type":"ContainerStarted","Data":"3b1bed1e3dd0f630f4ae82b402600cd808789b37bf65d6044ebd5cbccbfbb485"}
Nov 03 03:01:38.332617 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.332517    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb" event={"ID":"150b0a73-6cfc-438d-853f-3302f2232ec2","Type":"ContainerStarted","Data":"85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d"}
Nov 03 03:01:38.333461 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.333440    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-568md" event={"ID":"63321407-0b4f-4f4c-9f17-19aaf6ef081a","Type":"ContainerStarted","Data":"acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b"}
Nov 03 03:01:38.334445 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.334428    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rqm5c" event={"ID":"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9","Type":"ContainerStarted","Data":"bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a"}
Nov 03 03:01:38.814678 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:38.814639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:39.066726 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.066659    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:39.743663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.743620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:39.831794 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.831761    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:01:39.834663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.834639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.857520 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.857492    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:01:39.904220 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.904188    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 03:01:39.905643 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.905617    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 03:01:39.905790 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.905724    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 03:01:39.905790 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.905766    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 03:01:39.905902 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.905872    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 03:01:39.905980 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.905967    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 03:01:39.906053 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.906041    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 03:01:39.906120 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.906109    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 03:01:39.946541 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.946736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.946736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.946736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.946736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.946736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.946736 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.947000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.947000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.947000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.947000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.947000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.947000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.946909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:39.988364 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.988315    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x6qz9\""
Nov 03 03:01:39.988772 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:39.988747    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 03:01:40.047827 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.047827 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.047998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048094 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.048072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048419 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.048115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048419 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.048152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048419 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.048188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048419 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.048225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.048419 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.048381    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.050022 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.049780    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.050022 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.049976    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.051754 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.051654    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.052023 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.051997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055015 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.054993    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055205 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.055124    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055205 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.055064    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055205 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.055051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055205 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.055181    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055410 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.055224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.055410 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.055385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.111510 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.111477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.154728 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.154694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:40.345403 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.345309    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6k7g2" event={"ID":"996d2ead-9513-41c2-acd6-1ced1f624131","Type":"ContainerStarted","Data":"210dfea76c0b8b3086b11b74de0e49a24959fae5ea9b8cb9c619ee95ace70881"}
Nov 03 03:01:40.345403 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.345381    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6k7g2" event={"ID":"996d2ead-9513-41c2-acd6-1ced1f624131","Type":"ContainerStarted","Data":"ccb469821f3fefea9d373b9b0e9c1849d8295fbde544340c7666178cfbb2c801"}
Nov 03 03:01:40.467606 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.467280    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6k7g2" podStartSLOduration=65.929346261 podStartE2EDuration="1m8.467258359s" totalImagesPullingTime="2.537912098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:40.465257866 +0000 UTC m=+69.891990970" watchObservedRunningTime="2026-11-03 03:01:40.467258359 +0000 UTC m=+69.893991463"
Nov 03 03:01:40.528625 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.528581    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:40.528801 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.528686    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:40.534316 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.534291    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:40.645833 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:40.645803    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:01:40.649340 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:40.649306    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df4a68a_fd3f_4a36_9dda_f3b141bdec3f.slice/crio-64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc WatchSource:0}: Error finding container 64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc: Status 404 returned error can't find the container with id 64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc
Nov 03 03:01:41.351378 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.351336    2576 generic.go:356] "Generic (PLEG): container finished" podID="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" containerID="7a48b756d91375fccb796c297b10d915be6568847cdaaaea784b4057a06643d8" exitCode=0
Nov 03 03:01:41.351884 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.351424    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwq7p" event={"ID":"9e897c78-ce55-4d3f-bcfd-148c8faf01f2","Type":"ContainerDied","Data":"7a48b756d91375fccb796c297b10d915be6568847cdaaaea784b4057a06643d8"}
Nov 03 03:01:41.354340 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.354298    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc"}
Nov 03 03:01:41.359933 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.359909    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:41.746704 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.746659    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67c65cdb4b-65nnw"]
Nov 03 03:01:41.748914 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.748896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.820812 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.820779    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c65cdb4b-65nnw"]
Nov 03 03:01:41.866727 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.866696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.866727 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.866735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.866941 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.866776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.866941 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.866860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.866941 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.866891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.867031 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.866987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.867065 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.867052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.867096 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.867068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.874442 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.874401    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 03:01:41.934938 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.934897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:41.968244 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968432 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968432 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968432 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968432 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968432 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968738 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.968738 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.968463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.969153 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.969130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.969988 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.969961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.970131 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.970113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.970591 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.970544    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.970663 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.970633    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.975571 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.972661    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:41.975571 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:41.972844    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:42.035223 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.035157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:42.060240 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.060206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:42.158465 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.158430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:42.362678 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.362640    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwq7p" event={"ID":"9e897c78-ce55-4d3f-bcfd-148c8faf01f2","Type":"ContainerStarted","Data":"a8959781a24174fd7b8badec43c9c883539dc03d4f453ef6d433189a39c66655"}
Nov 03 03:01:42.363162 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.362742    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwq7p" event={"ID":"9e897c78-ce55-4d3f-bcfd-148c8faf01f2","Type":"ContainerStarted","Data":"8b053432b5df9f40c6e9b4723d664462bd91d3e5d5f9be5ae83fc2782556ad04"}
Nov 03 03:01:42.401501 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.401435    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:42.492682 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.492618    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wwq7p" podStartSLOduration=2.793479667 podStartE2EDuration="5.49260115s" totalImagesPullingTime="2.608882285s" totalInitContainerRuntime="90.239198ms" isStatefulPod=true podCreationTimestamp="2026-11-03 03:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:42.491809198 +0000 UTC m=+71.918542302" watchObservedRunningTime="2026-11-03 03:01:42.49260115 +0000 UTC m=+71.919334254"
Nov 03 03:01:42.500030 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.499996    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c65cdb4b-65nnw"]
Nov 03 03:01:42.502952 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:42.502906    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d44588_4826_47de_bd89_fa91a3ed291d.slice/crio-ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80 WatchSource:0}: Error finding container ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80: Status 404 returned error can't find the container with id ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80
Nov 03 03:01:42.586920 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.586054    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-875cc949b-lpm9j"]
Nov 03 03:01:42.588580 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.587940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.619437 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.619328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:42.643369 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.643337    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 03:01:42.643537 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.643415    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 03:01:42.646112 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.646084    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 03:01:42.646252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.646150    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-atn5vmneqg5d8\""
Nov 03 03:01:42.646252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.646186    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mrslv\""
Nov 03 03:01:42.646252 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.646237    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 03:01:42.654948 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.654860    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-875cc949b-lpm9j"]
Nov 03 03:01:42.674706 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.674670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-metrics-server-audit-profiles\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.674883 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.674802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.674883 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.674841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlvb\" (UniqueName: \"kubernetes.io/projected/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-kube-api-access-6qlvb\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.675000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.674913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-client-certs\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.675000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.674945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-tls\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.675000 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.674974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-audit-log\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.675155 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.675001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-client-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.775764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-tls\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.775822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-audit-log\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.775853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-client-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.775906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-metrics-server-audit-profiles\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.775970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.776002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlvb\" (UniqueName: \"kubernetes.io/projected/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-kube-api-access-6qlvb\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.776659 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.776085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-client-certs\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.778518 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.777536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-audit-log\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.778518 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.778057    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.779023 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.778980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-metrics-server-audit-profiles\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.780775 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.780741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-client-certs\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.782030 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.782005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-client-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.782276 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.782253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-tls\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.842788 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.842759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlvb\" (UniqueName: \"kubernetes.io/projected/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-kube-api-access-6qlvb\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:42.901723 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:42.901697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:43.012664 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.012629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:43.199817 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.199707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:43.339523 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.339465    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-875cc949b-lpm9j"]
Nov 03 03:01:43.342221 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:43.342190    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcff55ad_f5a9_4f59_afd0_ecae4aefa472.slice/crio-a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537 WatchSource:0}: Error finding container a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537: Status 404 returned error can't find the container with id a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537
Nov 03 03:01:43.367968 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.367929    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb" event={"ID":"150b0a73-6cfc-438d-853f-3302f2232ec2","Type":"ContainerStarted","Data":"2beffadd18a9604983c79860adfcb5cba9e116f13976e749a91855837d870868"}
Nov 03 03:01:43.369517 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.369492    2576 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="2186c7bcb7ea7471b917d29cbc78d35807c011607877376302f77d6ec6ef55d5" exitCode=0
Nov 03 03:01:43.369657 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.369579    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"2186c7bcb7ea7471b917d29cbc78d35807c011607877376302f77d6ec6ef55d5"}
Nov 03 03:01:43.370840 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.370793    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j" event={"ID":"dcff55ad-f5a9-4f59-afd0-ecae4aefa472","Type":"ContainerStarted","Data":"a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537"}
Nov 03 03:01:43.372500 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.372479    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c65cdb4b-65nnw" event={"ID":"e9d44588-4826-47de-bd89-fa91a3ed291d","Type":"ContainerStarted","Data":"648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788"}
Nov 03 03:01:43.372642 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.372526    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c65cdb4b-65nnw" event={"ID":"e9d44588-4826-47de-bd89-fa91a3ed291d","Type":"ContainerStarted","Data":"ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80"}
Nov 03 03:01:43.498698 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.498594    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb" podStartSLOduration=2.075122221 podStartE2EDuration="6.498579365s" totalImagesPullingTime="4.423457144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:43.49769779 +0000 UTC m=+72.924430894" watchObservedRunningTime="2026-11-03 03:01:43.498579365 +0000 UTC m=+72.925312463"
Nov 03 03:01:43.618439 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.618384    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67c65cdb4b-65nnw" podStartSLOduration=2.618366548 podStartE2EDuration="2.618366548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:43.617946163 +0000 UTC m=+73.044679295" watchObservedRunningTime="2026-11-03 03:01:43.618366548 +0000 UTC m=+73.045099651"
Nov 03 03:01:43.815167 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.815071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:43.989152 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:43.989103    2576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c65cdb4b-65nnw"]
Nov 03 03:01:44.023134 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.023058    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:44.097276 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.097181    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6776568cb9-jl5c5"]
Nov 03 03:01:44.100242 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.100218    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.172207 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.172168    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6776568cb9-jl5c5"]
Nov 03 03:01:44.188106 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188292 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188292 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188292 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188421 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188421 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188421 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.188421 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.188415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.289680 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.289645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.289837 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.289689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.289837 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.289810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.289934 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.289846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.289934 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.289892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290034 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290075 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290208 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290542 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290512    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290660 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290660 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290766 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.290954 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.290924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.292426 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.292405    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.292505 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.292440    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.360160 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.360127    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.415132 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.415096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:44.811498 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:44.811466    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6776568cb9-jl5c5"]
Nov 03 03:01:44.814770 ip-10-0-138-11 kubenswrapper[2576]: W1103 03:01:44.814737    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21108ceb_31ac_4158_bc6f_10553563603b.slice/crio-a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016 WatchSource:0}: Error finding container a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016: Status 404 returned error can't find the container with id a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016
Nov 03 03:01:45.322566 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:45.321684    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:45.383171 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:45.382821    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6776568cb9-jl5c5" event={"ID":"21108ceb-31ac-4158-bc6f-10553563603b","Type":"ContainerStarted","Data":"c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058"}
Nov 03 03:01:45.383171 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:45.382872    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6776568cb9-jl5c5" event={"ID":"21108ceb-31ac-4158-bc6f-10553563603b","Type":"ContainerStarted","Data":"a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016"}
Nov 03 03:01:45.715200 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:45.715146    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6776568cb9-jl5c5" podStartSLOduration=1.715134683 podStartE2EDuration="1.715134683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:45.71470142 +0000 UTC m=+75.141434524" watchObservedRunningTime="2026-11-03 03:01:45.715134683 +0000 UTC m=+75.141867783"
Nov 03 03:01:45.725832 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:45.725743    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:45.900745 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:45.900719    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:46.195683 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:46.195649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:46.389170 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:46.389097    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-568md" event={"ID":"63321407-0b4f-4f4c-9f17-19aaf6ef081a","Type":"ContainerStarted","Data":"eb62ffabc65087b9fbaf42ab8c3d61e47507bba187cf66300f458b84e738df8d"}
Nov 03 03:01:46.389781 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:46.389757    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:46.391763 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:46.391735    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j" event={"ID":"dcff55ad-f5a9-4f59-afd0-ecae4aefa472","Type":"ContainerStarted","Data":"d579e08f344d054b5fb37d03625d301468539cc1aeb3ab4453dd98b09410133f"}
Nov 03 03:01:46.508258 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:46.508210    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-568md" podStartSLOduration=66.402000091 podStartE2EDuration="1m14.508195554s" totalImagesPullingTime="8.106195463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:46.507064282 +0000 UTC m=+75.933797375" watchObservedRunningTime="2026-11-03 03:01:46.508195554 +0000 UTC m=+75.934928657"
Nov 03 03:01:46.629402 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:46.629333    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j" podStartSLOduration=1.838704522 podStartE2EDuration="4.629312009s" totalImagesPullingTime="2.790607487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:01:46.627227808 +0000 UTC m=+76.053960913" watchObservedRunningTime="2026-11-03 03:01:46.629312009 +0000 UTC m=+76.056045112"
Nov 03 03:01:47.312965 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.312931    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:47.399811 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.399771    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"03fdc0ca8e31d21afe5b3c49458f95638f1b5f283355d292ff123cb11a5324da"}
Nov 03 03:01:47.399991 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.399860    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"cc8a8b6721db0a52bb5f5a8904e9b91ed00393b312fce35f98031512a3d5d4b3"}
Nov 03 03:01:47.399991 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.399902    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"1e9c519f23c24214c9fd6967289cdca1a56fc323058c45286ff16b85a4cdf21b"}
Nov 03 03:01:47.399991 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.399939    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"5c7e0795e9abdf01874a6f9577b6fbcc52d85bfdec919a2e40b2bf4124a749cb"}
Nov 03 03:01:47.399991 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.399989    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"e22567b2d9dc32fc9530ed14219acbb3e80ebd054e11e5fa762bb6b93617d455"}
Nov 03 03:01:47.553815 ip-10-0-138-11 kubenswrapper[2576]: I1103 03:01:47.553691    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:48.709921 ip-10-0-138-11 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 03:01:48.731279 ip-10-0-138-11 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 03:01:48.731466 ip-10-0-138-11 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 03:01:48.731533 ip-10-0-138-11 systemd[1]: kubelet.service: Consumed 4.478s CPU time.
Nov 03 03:01:48.733962 ip-10-0-138-11 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 03:01:48.800881 ip-10-0-138-11 kubenswrapper[5515]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 03:01:48.800881 ip-10-0-138-11 kubenswrapper[5515]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 03:01:48.800881 ip-10-0-138-11 kubenswrapper[5515]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 03:01:48.800881 ip-10-0-138-11 kubenswrapper[5515]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 03:01:48.802940 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802852    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.802940 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802940    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.802940 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802944    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802947    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802950    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802953    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802956    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802958    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802961    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802964    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802966    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802969    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802972    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802975    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802977    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802980    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802983    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802985    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802988    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802991    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.802998    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803001    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.803038 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803004    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803007    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803010    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803012    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803021    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803026    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803028    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803032    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803035    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803038    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803040    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803043    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803046    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803049    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803052    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803054    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803057    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803060    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803062    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803065    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.803577 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803067    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803070    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803072    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803075    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803077    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803080    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803082    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803085    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803087    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803090    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803093    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803096    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803099    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803102    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803105    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803110    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803113    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803116    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803119    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.804093 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803122    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803124    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803127    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803129    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803132    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803134    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803137    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803140    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803142    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803145    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803147    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803151    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803154    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803156    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803159    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803161    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803164    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803167    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803169    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.804620 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803172    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803175    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803177    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803180    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803182    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803185    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803188    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803190    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803193    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803196    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803198    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803200    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803204    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803206    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803209    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803211    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803213    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803216    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803218    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.805101 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803221    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803223    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803226    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803228    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803231    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803233    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.803236    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804004    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804012    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804015    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804018    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804020    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804023    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804026    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804030    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804032    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804035    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804039    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804043    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.805638 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804047    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804050    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804053    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804056    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804058    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804061    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804064    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804067    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804070    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804072    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804075    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804077    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804080    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804082    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804085    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804087    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804090    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804093    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804095    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.806144 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804098    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804101    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804103    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804106    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804109    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804112    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804114    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804117    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804119    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804122    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804125    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804127    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804131    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804137    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804140    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804143    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804145    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804148    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804150    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804153    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.806670 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804155    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804158    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804162    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804164    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804167    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804170    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804172    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804175    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804178    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804180    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804183    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804185    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804188    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804191    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804194    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804197    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804199    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804202    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804204    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804207    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.807164 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804211    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804213    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804216    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804218    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804221    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804224    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804226    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804229    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804231    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804234    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804236    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804239    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804241    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804244    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804247    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804250    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804252    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804255    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804257    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.807711 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804260    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804262    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804265    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804267    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804270    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804273    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804275    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804278    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804280    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804283    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804285    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804288    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804291    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804294    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804296    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804299    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804843    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804849    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804852    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804855    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.808212 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804858    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804860    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804863    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804866    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804868    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804871    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804874    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804881    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804883    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804887    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804889    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804892    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804894    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804897    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804900    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804902    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804905    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804907    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804910    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804913    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.808749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804915    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804918    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804920    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804924    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804927    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804929    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804932    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804935    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804937    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804940    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804942    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804945    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804948    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804950    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804953    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804956    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804959    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804961    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804964    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804967    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.809284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804970    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804972    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804975    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804977    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804980    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804983    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804985    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804988    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804990    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804993    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804995    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.804998    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805001    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805004    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805006    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805009    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805012    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805014    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805017    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805019    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.810297 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805022    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805024    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805027    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805030    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805033    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805035    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805037    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805040    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805042    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805045    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805048    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805050    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805053    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805056    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805059    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805061    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805064    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805066    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805068    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805071    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.810941 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805074    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805076    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805078    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805081    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805084    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805086    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805089    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805091    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805094    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805096    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805101    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805105    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805110    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805114    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805116    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805119    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805121    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805126    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805130    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.811452 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805133    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805136    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805139    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805236    5515 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805247    5515 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805253    5515 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805257    5515 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805262    5515 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805266    5515 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805270    5515 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805282    5515 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805285    5515 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805288    5515 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805292    5515 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805295    5515 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805298    5515 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805301    5515 flags.go:64] FLAG: --cgroup-root=""
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805304    5515 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805307    5515 flags.go:64] FLAG: --client-ca-file=""
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805310    5515 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805313    5515 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805318    5515 flags.go:64] FLAG: --cluster-domain=""
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805321    5515 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805324    5515 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 03:01:48.811968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805327    5515 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805332    5515 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805337    5515 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805340    5515 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805343    5515 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805347    5515 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805350    5515 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805353    5515 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805356    5515 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805359    5515 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805362    5515 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805366    5515 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805369    5515 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805372    5515 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805375    5515 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805378    5515 flags.go:64] FLAG: --enable-server="true"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805381    5515 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805385    5515 flags.go:64] FLAG: --event-burst="100"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805388    5515 flags.go:64] FLAG: --event-qps="50"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805391    5515 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805394    5515 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805397    5515 flags.go:64] FLAG: --eviction-hard=""
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805401    5515 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805403    5515 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805406    5515 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 03:01:48.812568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805409    5515 flags.go:64] FLAG: --eviction-soft=""
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805412    5515 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805415    5515 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805418    5515 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805421    5515 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805424    5515 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805427    5515 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805429    5515 flags.go:64] FLAG: --feature-gates=""
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805435    5515 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805438    5515 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805441    5515 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805444    5515 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805447    5515 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805450    5515 flags.go:64] FLAG: --help="false"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805453    5515 flags.go:64] FLAG: --hostname-override="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805456    5515 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805459    5515 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805462    5515 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805465    5515 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805468    5515 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805471    5515 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805474    5515 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805477    5515 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 03:01:48.813199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805480    5515 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805484    5515 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805487    5515 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805490    5515 flags.go:64] FLAG: --kube-reserved=""
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805493    5515 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805496    5515 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805499    5515 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805502    5515 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805504    5515 flags.go:64] FLAG: --lock-file=""
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805507    5515 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805510    5515 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805513    5515 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805518    5515 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805521    5515 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805524    5515 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805528    5515 flags.go:64] FLAG: --logging-format="text"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805531    5515 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805535    5515 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805538    5515 flags.go:64] FLAG: --manifest-url=""
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805541    5515 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805564    5515 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805567    5515 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805571    5515 flags.go:64] FLAG: --max-pods="110"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805574    5515 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805578    5515 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 03:01:48.813795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805580    5515 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805583    5515 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805586    5515 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805589    5515 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805592    5515 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805599    5515 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805602    5515 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805605    5515 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805608    5515 flags.go:64] FLAG: --pod-cidr=""
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805612    5515 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805615    5515 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805618    5515 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805620    5515 flags.go:64] FLAG: --port="10250"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805623    5515 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805626    5515 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02cf534ab24e44e9e"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805629    5515 flags.go:64] FLAG: --qos-reserved=""
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805632    5515 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805635    5515 flags.go:64] FLAG: --register-node="true"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805638    5515 flags.go:64] FLAG: --register-with-taints=""
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805642    5515 flags.go:64] FLAG: --registry-burst="10"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805645    5515 flags.go:64] FLAG: --registry-qps="5"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805648    5515 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805651    5515 flags.go:64] FLAG: --reserved-memory=""
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805655    5515 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805659    5515 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805662    5515 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 03:01:48.814413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805665    5515 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805668    5515 flags.go:64] FLAG: --runonce="false"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805671    5515 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805674    5515 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805677    5515 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805679    5515 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805682    5515 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805685    5515 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805691    5515 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805694    5515 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805697    5515 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805699    5515 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805702    5515 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805705    5515 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805708    5515 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805711    5515 flags.go:64] FLAG: --system-cgroups=""
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805714    5515 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805719    5515 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805722    5515 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805725    5515 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805729    5515 flags.go:64] FLAG: --tls-min-version=""
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805731    5515 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805734    5515 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805737    5515 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805740    5515 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 03:01:48.815091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805743    5515 flags.go:64] FLAG: --v="2"
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805747    5515 flags.go:64] FLAG: --version="false"
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805751    5515 flags.go:64] FLAG: --vmodule=""
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805755    5515 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.805759    5515 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805889    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805893    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805896    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805899    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805902    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805904    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805907    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805909    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805912    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805914    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805918    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805921    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805923    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805926    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805928    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.815872 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805931    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805934    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805937    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805939    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805942    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805945    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805947    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805950    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805952    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805955    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805957    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805960    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805962    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805965    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805968    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805972    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805974    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805978    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805981    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.816500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805983    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805986    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805988    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805991    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805993    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805996    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.805998    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806001    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806004    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806007    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806010    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806012    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806015    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806025    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806028    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806030    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806033    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806035    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806038    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.817373 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806041    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806043    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806046    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806048    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806051    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806053    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806056    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806059    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806061    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806064    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806066    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806070    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806073    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806075    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806078    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806080    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806083    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806086    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806088    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.817921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806091    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806093    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806097    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806099    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806102    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806104    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806107    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806110    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806176    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806180    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806182    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806185    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806188    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806191    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806196    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806200    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806203    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806206    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806209    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806212    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.818410 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806214    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806217    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806219    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806222    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806224    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806229    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806231    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806234    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806236    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806239    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806242    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806244    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806247    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806249    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.806281    5515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806378    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.819001 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806384    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806387    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806390    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806393    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806396    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806401    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806403    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806406    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806408    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806411    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806414    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806417    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806419    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806422    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806425    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806427    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806431    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806433    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806436    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.819519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806438    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806441    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806444    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806446    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806449    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806451    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806454    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806457    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806459    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806462    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806465    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806467    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806471    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806474    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806476    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806479    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806482    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806484    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806488    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806490    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.820127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806493    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806496    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806498    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806501    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806505    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806509    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806513    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806516    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806518    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806520    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806523    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806526    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806528    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806531    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806533    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806536    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806538    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806541    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806543    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806564    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.820648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806568    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806572    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806576    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806578    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806582    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806585    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806588    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806590    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806593    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806596    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806599    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806602    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806605    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806607    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806610    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806612    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806615    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806618    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806621    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.821129 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806624    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806626    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806629    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806631    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806634    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806637    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806639    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806642    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806644    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806647    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806649    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806652    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806654    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806657    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806662    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806665    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806668    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806673    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806676    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.821615 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806679    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806682    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806684    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806687    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806691    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806693    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806696    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.806698    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.811682    5515 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.811700    5515 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.811721    5515 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.811726    5515 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811786    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811792    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811795    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811798    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811800    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811804    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811806    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.822077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811809    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811812    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811814    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811817    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811819    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811823    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811825    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811828    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811834    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811837    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811840    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811844    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811847    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811851    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811854    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811857    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811860    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811863    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811867    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.822605 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811871    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811874    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811877    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811880    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811883    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811885    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811888    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811890    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811893    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811896    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811898    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811901    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811903    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811906    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811908    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811911    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811914    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811916    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811919    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811921    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.823076 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811924    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811927    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811930    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811932    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811935    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811937    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811941    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811943    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811946    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811948    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811951    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811953    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811965    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811968    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811971    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811974    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811977    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811980    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811982    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811986    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.823674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811988    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811991    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811993    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811996    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.811998    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812001    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812003    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812006    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812008    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812011    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812014    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812016    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812019    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812022    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812025    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812027    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812030    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812032    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812035    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812038    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.824379 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812041    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812044    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812046    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812049    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812051    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812054    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812056    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812059    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812061    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812064    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812066    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812069    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812072    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812074    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812077    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812079    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812082    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812084    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812087    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.825176 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812090    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.812118    5515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812254    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812258    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812261    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812264    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812266    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812270    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812272    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812275    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812277    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812280    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812282    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812285    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812288    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812290    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.825895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812293    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812295    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812298    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812301    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812303    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812306    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812308    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812311    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812313    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812316    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812318    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812321    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812323    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812326    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812329    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812331    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812334    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812337    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812339    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812342    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.826795 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812344    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812346    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812349    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812351    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812355    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812358    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812360    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812363    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812366    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812368    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812371    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812373    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812376    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812378    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812382    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812385    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812387    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812390    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812392    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.827855 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812395    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812397    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812400    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812403    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812405    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812407    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812410    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812412    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812415    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812418    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812421    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812423    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812425    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812428    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812430    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812433    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812436    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812438    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812441    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812444    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.828741 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812446    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812449    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812451    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812454    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812456    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812459    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812461    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812464    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812466    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812469    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812471    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812474    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812477    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812479    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812481    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812484    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812486    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812489    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812491    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812494    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.829765 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812496    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812499    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812501    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812504    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812506    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812509    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812511    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812514    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812517    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812519    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812523    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812528    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812532    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.812571    5515 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812605    5515 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812610    5515 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 03:01:48.830870 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812613    5515 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812616    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812619    5515 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812622    5515 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812625    5515 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812628    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812630    5515 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812633    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812635    5515 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812638    5515 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812641    5515 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812643    5515 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812646    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812648    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812651    5515 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812656    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812659    5515 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812663    5515 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812666    5515 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 03:01:48.831679 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812669    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812672    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812674    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812677    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812680    5515 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812682    5515 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812685    5515 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812690    5515 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812692    5515 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812696    5515 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812699    5515 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812701    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812704    5515 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812707    5515 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812709    5515 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812712    5515 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812714    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812717    5515 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812719    5515 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 03:01:48.832539 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812722    5515 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812725    5515 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812727    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812730    5515 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812732    5515 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812735    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812737    5515 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812740    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812742    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812746    5515 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812748    5515 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812751    5515 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812753    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812756    5515 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812759    5515 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812762    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812765    5515 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812767    5515 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812770    5515 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812772    5515 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 03:01:48.833564 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812776    5515 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812779    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812782    5515 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812785    5515 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812788    5515 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812790    5515 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812793    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812795    5515 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812798    5515 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812800    5515 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812803    5515 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812805    5515 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812808    5515 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812810    5515 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812814    5515 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812817    5515 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812820    5515 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812823    5515 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812825    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812828    5515 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 03:01:48.834371 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812830    5515 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812834    5515 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812837    5515 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812839    5515 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812842    5515 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812845    5515 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812847    5515 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812850    5515 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812852    5515 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812855    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812857    5515 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812860    5515 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812864    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812868    5515 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812870    5515 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812873    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812876    5515 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812879    5515 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812881    5515 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 03:01:48.835234 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812884    5515 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812886    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812889    5515 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812891    5515 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812894    5515 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812896    5515 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:01:48.812900    5515 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.812961    5515 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814000    5515 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814057    5515 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814226    5515 server.go:1014] "Starting client certificate rotation"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814370    5515 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814418    5515 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 02:55:32 +0000 UTC" deadline="2028-05-27 13:17:00.190333528 +0000 UTC"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814454    5515 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13714h15m11.375881586s"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.814715    5515 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.815832    5515 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 03:01:48.835825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.817057    5515 log.go:25] "Validated CRI v1 runtime API"
Nov 03 03:01:48.836250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.818880    5515 log.go:25] "Validated CRI v1 image API"
Nov 03 03:01:48.836250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.819656    5515 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 03:01:48.836250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.821720    5515 fs.go:135] Filesystem UUIDs: map[3814681a-6274-4549-bd24-b3a06cd3952b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c903799d-0ba2-4815-bce0-8348f9bc9138:/dev/nvme0n1p3]
Nov 03 03:01:48.836413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.821740    5515 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~projected/kube-api-access-9sl9g:{mountpoint:/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~projected/kube-api-access-9sl9g major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~projected/kube-api-access-6hcn5:{mountpoint:/var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~projected/kube-api-access-6hcn5 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~projected/kube-api-access-fjsrx:{mountpoint:/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~projected/kube-api-access-fjsrx major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35125e2e-9a2d-4577-98ca-5966cfe1d3a1/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/35125e2e-9a2d-4577-98ca-5966cfe1d3a1/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44942777-b4b6-4264-aeff-7c88943cf14e/volumes/kubernetes.io~projected/kube-api-access-7v9dm:{mountpoint:/var/lib/kubelet/pods/44942777-b4b6-4264-aeff-7c88943cf14e/volumes/kubernetes.io~projected/kube-api-access-7v9dm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63321407-0b4f-4f4c-9f17-19aaf6ef081a/volumes/kubernetes.io~projected/kube-api-access-bnz9k:{mountpoint:/var/lib/kubelet/pods/63321407-0b4f-4f4c-9f17-19aaf6ef081a/volumes/kubernetes.io~projected/kube-api-access-bnz9k major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64651a3d-f058-40c3-9171-53ba4e35d04a/volumes/kubernetes.io~projected/kube-api-access-5skpl:{mountpoint:/var/lib/kubelet/pods/64651a3d-f058-40c3-9171-53ba4e35d04a/volumes/kubernetes.io~projected/kube-api-access-5skpl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~projected/kube-api-access-rdb6f:{mountpoint:/var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~projected/kube-api-access-rdb6f major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a2050bd-da88-4923-b686-4c9e70226b59/volumes/kubernetes.io~projected/kube-api-access-xxb44:{mountpoint:/var/lib/kubelet/pods/7a2050bd-da88-4923-b686-4c9e70226b59/volumes/kubernetes.io~projected/kube-api-access-xxb44 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81335b4a-183a-49d7-81d0-8bff20d94e77/volumes/kubernetes.io~projected/kube-api-access-kx4f4:{mountpoint:/var/lib/kubelet/pods/81335b4a-183a-49d7-81d0-8bff20d94e77/volumes/kubernetes.io~projected/kube-api-access-kx4f4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/alertmanager/9 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/config-reloader/6 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~projected/kube-api-access-knr78:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~projected/kube-api-access-knr78 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~projected/tls-assets major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/config-volume major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/web-config major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/996d2ead-9513-41c2-acd6-1ced1f624131/volumes/kubernetes.io~projected/kube-api-access-xdgk8:{mountpoint:/var/lib/kubelet/pods/996d2ead-9513-41c2-acd6-1ced1f624131/volumes/kubernetes.io~projected/kube-api-access-xdgk8 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/996d2ead-9513-41c2-acd6-1ced1f624131/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/996d2ead-9513-41c2-acd6-1ced1f624131/volumes/kubernetes.io~secret/metrics-certs major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~projected/kube-api-access-rpls9:{mountpoint:/var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~projected/kube-api-access-rpls9 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volumes/kubernetes.io~projected/kube-api-access-xhbdq:{mountpoint:/var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volumes/kubernetes.io~projected/kube-api-access-xhbdq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc29a623-58c0-41a7-9167-926cc8e5b8c8/volumes/kubernetes.io~projected/kube-api-access-tfwr2:{mountpoint:/var/lib/kubelet/pods/bc29a623-58c0-41a7-9167-926cc8e5b8c8/volumes/kubernetes.io~projected/kube-api-access-tfwr2 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc29a623-58c0-41a7-9167-926cc8e5b8c8/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/bc29a623-58c0-41a7-9167-926cc8e5b8c8/volumes/kubernetes.io~secret/metrics-tls major:0 minor:123 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7d44eda-e0e2-4bea-afeb-160eebdc1710/volumes/kubernetes.io~projected/kube-api-access-dsp96:{mountpoint:/var/lib/kubelet/pods/c7d44eda-e0e2-4bea-afeb-160eebdc1710/volumes/kubernetes.io~projected/kube-api-access-dsp96 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7d44eda-e0e2-4bea-afeb-160eebdc1710/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c7d44eda-e0e2-4bea-afeb-160eebdc1710/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~projected/kube-api-access-6qlvb:{mountpoint:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~projected/kube-api-access-6qlvb major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~projected/kube-api-access-b4wcj:{mountpoint:/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~projected/kube-api-access-b4wcj major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f023c708-c71a-4dcb-99cd-f604cb0088ce/volumes/kubernetes.io~projected/kube-api-access-2ptbp:{mountpoint:/var/lib/kubelet/pods/f023c708-c71a-4dcb-99cd-f604cb0088ce/volumes/kubernetes.io~projected/kube-api-access-2ptbp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f023c708-c71a-4dcb-99cd-f604cb0088ce/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f023c708-c71a-4dcb-99cd-f604cb0088ce/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09cc1ed-086f-4b12-a9ff-e465fbe94038/volumes/kubernetes.io~projected/kube-api-access-8d5vf:{mountpoint:/var/lib/kubelet/pods/f09cc1ed-086f-4b12-a9ff-e465fbe94038/volumes/kubernetes.io~projected/kube-api-access-8d5vf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff3b7b52-b0ad-4965-ba72-39330944d766/volumes/kubernetes.io~projected/kube-api-access-v9kdd:{mountpoint:/var/lib/kubelet/pods/ff3b7b52-b0ad-4965-ba72-39330944d766/volumes/kubernetes.io~projected/kube-api-access-v9kdd major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ef71982f9b2cd2072909ae898a211a28a373e7410c0bb70a73670e22a9917549/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5141b1962ffd60ff2be9654407ed9f900dcfcd9229fa09df5e04c99adf11c173/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/85a29ba3ae225fbfc30312bb0715b25c6b3dcea61dd820c1041140a878519a08/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f88081ef1689a6dcae6e24cf800cae1dd5ee2edd2ef18f24257a894e47a13956/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/734187cdcd0a96468c76b69a71f9e5f2a974e2cd848182946c346aa0fff0b9f1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7c11aa11c1d3cf1980970348323f50c25050a2b8ece2ec76b56f71391d5b4845/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/d060c0b94c15a8326fb444bd1c135898f1f390c4dccac45bf525b46d41c664e7/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9b7afcbd399144500b6ca1fa33dd29a782a299a142c015569bf3708a189d4227/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/61293c3e031942dc72aa011e4fbbf00ac80ae957b1e60736a5460e8f224ec424/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/edec90b3206e9905567875d27419dcca375c9510499c4089834779acfdfa8ad2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/9cb056ebc879ed8bdc3c3e836a607c5e184b67e62e4b80b724a5131ad38dd750/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/56cdda17d70621c0fdbcd4d346965d469c44f09e2fb61599c1f8696e2140e725/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/14eea497312f9f11e25135345ec6ce9225cd8faada1754a2876f4cee9fcd419e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f58173c2f1bce917f2e402b5792d52ddcfd38db56f856c39c2c0dc3eed6f407a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ac60200db07f07f8529539ae49e028532b3fa61f008aef730f81b42c7dbdd1d4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e3a64f6d6a62c932254dfbc14fb436c6a0ec8dc899e0f713b6d0f48c8e75209f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/76dc770dc7502e24afe5d54fbb8973e33e68e155ea369fbfdffc44d594420b95/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/56bc82af946e80770e3b390a11fe2c24d8d711bb92f281eab4eaf473c764a9a8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/3d4fce078724be83cec807d8f1909ddfb196369c41e154d425e2063a43a62b1b/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-195:{mountpoint:/var/lib/containers/storage/overlay/82784959f2373299105debaf0c2303772eb34a2c953828e988c9c0ad42af18c2/merged major:0 minor:195 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/6c1f09131c0309f0295a9a0288f1c27c07f7ac32c18ef9fc1019fe1f8ca7f169/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3adf122fde02d647dee6f1379d3c2439a9c0ca5d273892f2d8d5571e6c01d894/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2fb77bbdc100ed546804a51d3f135989cbc0d47b697ca304d397bfe36464cc70/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/5051a00303067e2c47d413bb880fb96dbeae6a105d2fd30a3310d279dae9aa3b/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/2f4d16453150fb2bdbf02f9e3345620e44395abc3e13e3b50bc46fe7954c30c0/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e84a4117b66c98fcf108f82dfeddec4748bb709f0c0de34ea60a2fc668398741/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/779cc9edb0e6028f77e2d2b81b92e77a01ef4ad4c0516a1cdfdc26991fcca5cb/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/fc766b4dff9c7e210498141b2bc95006d1f866686c89f2b31292795282bd235a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/a79ab57d90645abf9f184140d42ebe468919a92b5e18d1b4d0f5d7eaf1a5a406/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/629dd50f7a3dd0bdc2b0e392266edbc80600fd887396efb3b5449253e59974bc/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/6183c4da197e43b7d7c3137411dadac1f87e882ef6fcbe08050ab76cce9720ce/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/73b5888a92810b3ad15b498f3ef2ae6982742198ad1da031f5b77c4caaf0d591/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/9dd932b6d13af0e88d295062531e614c4c9dc1e919da6225844054ff09611b01/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/45f6a283a2761bc2fe2a3f5c59a32a3cf5a162060a1a411818f207d9a893b446/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/97ce4d5f8f474a772504861f36f4b78943521760458ae4ad20c9f85bb690bb32/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/8ceefedf18490d150c6ec800c21517c5fb9d76febe21958e4685edaffb7fa490/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/fd7dee99dbdc93ce6e47566b65adbe6bf04f275780ddd4a4347406bdd7f911c3/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/856417a58539e2ede94e32467cc78ceef6d4fcc3cd186e8cc4f9e71f3de1a8ef/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/833efce6764ffc319a8a6f87cdb1dd44b08b91971627c2645e32eb0c5165b741/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/410f2c5c0958a2e4fae6e3b63e57a9a980fe68e32947a083f38e12c5180ef6e5/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/80fdee4177d45c75161e967bef5283659a639ffd9628e982b061e8b910425572/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/05666811cf204e2dd9d94d1af550dab7a467acde572b56b716c5a683746d57a3/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/c8bbd8415f68883d1335afae8899e7d138c712d6532c41c5a04dc9460632aa5c/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/9f76695fcc3a98ccc5f70d6d93a80b5397381f6dad9bfbcc970ceb538de52874/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/0f3d826d018fc5a9fa77a7785dde8dea936634bf414e406eb263642d7152adcb/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/2a53b3bef80c3909e64c19a685d5309c80cc930cbac8eca2df7b969b6ef30d5d/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/9584b4f19c93bc03da836aeb407a7619e4f161fba3303a525e8c24b9b10da83a/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/f02d3e9a8bc33c29834453fdfbab2452f59937fd4ca3c6726a608f5c0a64fd8d/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/afc022e9f49765c780fc6318161d46cdf6b1fd007b914cb3153525b8a7e6c068/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/81edd7123b1c25fffe134d1ed27d472939a4e706661fc1ebf3e2a902a5d74d65/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/8d40ed8de3baa40efc4f38bed7a01cce6625cd42008e0da883baeb1ec1e6826a/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/a6354e3e4364c3939cb3ab5446a39a5ad8057679dc64b1edae8581a58adb83ea/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/a6a23244734b307f24433ad1725096f2afb29457e4dda6142a1d02fafa32525d/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/b8c1694e242326096f396b8f4254eb4bb80d261981af9e176e4e5a6b9151bc3e/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/85a81449db81bfa583e4bbbcd3f5a3240842f74f857ea074da7313005067357e/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/f1b0a4e4a1094bd99f5d5692e2a12872e5c3ac4a0aeabdf31e0e308503ac33e6/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/25c4f79bb85ef58285497092d1b767b1fe755ebf6f54955782f6f5288be3846a/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/6932f98a959ac527873538c87c826c4d2258131779e846223bdbf5129899dada/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/db1f094e96ae4f86a43884226a1df81a1f1d902d79d611f22a8bc5fcbac90e38/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d47b99b8c2c863aa7dc8d5fb9c9e878efae3a6816f7a2c692f8af796a7d6902d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1ee9f3ae89fb770ebab7de0e7e890c932d4bf8063b7991a6a3100dec46b17c68/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/815b96e7efe80f139bc2aff72f0ae1d376718de0d31d5e78e0c083ed5a29efa3/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/779bf08a1da03c4e4114ce773adcd829f65b941599e0f5b9c0e05f289a5e9914/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/997f4ee4d850669804489da0dd3ba34745412dff44d141527f844bf3d0501387/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/fc97bcdff42a5e6f76bb6518e18b518a29a1c1d8c88bc648576827e949633ea4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fd52a375d2db2fde76afef62af5a50eae4dd939915a3e819c92f728201d6f6e0/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/fde3248f69bb10923dca6e4c98b1b0dfd2110772f0d37a9712c3c3bd525eee3b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a6d7303fd11691d11eb1dccc29dc782c2197a71225f6b507f848639593590b59/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9ec0745771622a9b6f8b239d23fc204189058d0aad29e15d51313d37b8ac6a2d/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.832119    5515 manager.go:217] Machine: {Timestamp:2026-11-03 03:01:48.831251973 +0000 UTC m=+0.062144383 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3125324 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e34d048943c3fb7c22ff26aee64b9 SystemUUID:ec2e34d0-4894-3c3f-b7c2-2ff26aee64b9 BootID:4e5ef439-5727-455e-8fc3-6014b90deb7e Filesystems:[{Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:364 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ff3b7b52-b0ad-4965-ba72-39330944d766/volumes/kubernetes.io~projected/kube-api-access-v9kdd DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-195 DeviceMajor:0 DeviceMinor:195 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volumes/kubernetes.io~projected/kube-api-access-xhbdq DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61bb3a7722c6d06622df3a903d1366e3bc8c6b013cb506a6a893c1a1e0cccbf1/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f023c708-c71a-4dcb-99cd-f604cb0088ce/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acbaf166bbe51e8b70db2074f09a253b76b38f54c5cd0995fae17e7990c7018b/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~projected/kube-api-access-knr78 DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ca1d06b299ae6ee47805ed7300d1417f3730acbf07d54a4b0b5ab65036882d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/284adc92f992ae1dba4a7c8dbe9a770f479c214d727570e062962d8093e30bf0/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35125e2e-9a2d-4577-98ca-5966cfe1d3a1/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7a2050bd-da88-4923-b686-4c9e70226b59/volumes/kubernetes.io~projected/kube-api-access-xxb44 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a98001ec72d1047db9147c4e301cbf67a9656f16ddc241d6a7cbe6bf30aa336d/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44942777-b4b6-4264-aeff-7c88943cf14e/volumes/kubernetes.io~projected/kube-api-access-7v9dm DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~projected/kube-api-access-fjsrx DeviceMajor:0 DeviceMinor:366 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04cf5c36718806d5cc8d9161a800f8ff07b16f08395f67127c37bba564d5a97a/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7d44eda-e0e2-4bea-afeb-160eebdc1710/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/996d2ead-9513-41c2-acd6-1ced1f624131/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fdebfba47891c98d0da644f82afc6a04d52ffe3ffeafc9d5fda392f0ea93338/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~projected/kube-api-access-6qlvb DeviceMajor:0 DeviceMinor:359 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85136617c0c16ff494a511b27491feb0c1c772dc6ceba01aa65c7b080475460d/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6d8c81b844125c4ec54ed0436b836db8f1126b904504c1fc47204df9457f00f/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc29a623-58c0-41a7-9167-926cc8e5b8c8/volumes/kubernetes.io~projected/kube-api-access-tfwr2 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~projected/kube-api-access-rdb6f DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63321407-0b4f-4f4c-9f17-19aaf6ef081a/volumes/kubernetes.io~projected/kube-api-access-bnz9k DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f09cc1ed-086f-4b12-a9ff-e465fbe94038/volumes/kubernetes.io~projected/kube-api-access-8d5vf DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dec7dbbd4e00331844cc7aa40b2240a407d87a511fc6796af9cf9630ed11b0d/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:358 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbabfd1834576f1fcb505bcc3f18b0fbe19683d970118446d9fd46a48635f24d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes/kubernetes.io~projected/kube-api-access-9sl9g DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf2a68e1dd2fdd95b0fcc370df55673f6bc2da9fd3b51456fd298732f063251a/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2bd3b0861282b198da8800657d794e7a5aa87f4dee3011c62bac73644881e34/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c89eecf145fcd9cd8adca98bea9757b79f762502e5142171376b359c170a0f3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dcff55ad-f5a9-4f59-afd0-ecae4aefa472/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/996d2ead-9513-41c2-acd6-1ced1f624131/volumes/kubernetes.io~projected/kube-api-access-xdgk8 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f023c708-c71a-4dcb-99cd-f604cb0088ce/volumes/kubernetes.io~projected/kube-api-access-2ptbp DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/150b0a73-6cfc-438d-853f-3302f2232ec2/volumes/kubernetes.io~projected/kube-api-access-6hcn5 DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:281 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:304 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e897c78-ce55-4d3f-bcfd-148c8faf01f2/volumes/kubernetes.io~projected/kube-api-access-rpls9 DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e7ec81ad98d0ae5b3ff6421b790111d06499bdf95fdd4a3f97f208c6a4cfd8f/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc29a623-58c0-41a7-9167-926cc8e5b8c8/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:123 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/593b2de46dfac97db101698ec3fa01d35ed26c2d98ba0871d28c4b861108da0b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes/kubernetes.io~projected/kube-api-access-b4wcj DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/794ed6bd-5ce9-4aea-99ab-75607b2cd846/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81335b4a-183a-49d7-81d0-8bff20d94e77/volumes/kubernetes.io~projected/kube-api-access-kx4f4 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:365 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c603e8-5a0a-42af-a866-3439e1fdef56/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7d44eda-e0e2-4bea-afeb-160eebdc1710/volumes/kubernetes.io~projected/kube-api-access-dsp96 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3c47ea26844d8a50203e39a8549fe37bd409350e4414c423a98b866f4cee537/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64651a3d-f058-40c3-9171-53ba4e35d04a/volumes/kubernetes.io~projected/kube-api-access-5skpl DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04cf5c36718806d MacAddress:ba:ec:2d:fe:d3:27 Speed:10000 Mtu:8901} {Name:284adc92f992ae1 MacAddress:f6:2d:1e:d0:41:84 Speed:10000 Mtu:8901} {Name:2c89eecf145fcd9 MacAddress:32:56:2b:e2:a3:a3 Speed:10000 Mtu:8901} {Name:593b2de46dfac97 MacAddress:9e:b2:0c:12:e8:63 Speed:10000 Mtu:8901} {Name:64cb7b26e91109f MacAddress:0a:d8:37:38:ce:94 Speed:10000 Mtu:8901} {Name:85136617c0c16ff MacAddress:52:e7:f0:36:4f:6b Speed:10000 Mtu:8901} {Name:a3c47ea26844d8a MacAddress:fe:c2:52:93:ac:66 Speed:10000 Mtu:8901} {Name:a836549baf38978 MacAddress:56:ae:68:d6:6a:b4 Speed:10000 Mtu:8901} {Name:acbaf166bbe51e8 MacAddress:4e:76:fa:1b:8b:9f Speed:10000 Mtu:8901} {Name:bf2a68e1dd2fdd9 MacAddress:d6:ec:0b:cd:6b:02 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:38:8c:80:ab:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:1a:df:ad:d6:1f Speed:0 Mtu:8901} {Name:ef70771c705eca2 MacAddress:a2:4c:a2:1a:d5:99 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:38:8c:80:ab:ef Speed:0 Mtu:9001} {Name:ff71657a72c1533 MacAddress:c2:c5:4c:87:bd:e9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e6:8f:df:79:90:dd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:ab:20:4b:51:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.832788    5515 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.832857    5515 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833193    5515 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833219    5515 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-11.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833436    5515 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833447    5515 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833462    5515 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833485    5515 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833650    5515 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.833915    5515 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.834060    5515 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.834106    5515 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.834120    5515 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.834136    5515 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.834150    5515 kubelet.go:405] "Adding apiserver pod source"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.834162    5515 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.835060    5515 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.835073    5515 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 03:01:48.836784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.835659    5515 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 03:01:48.839100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839085    5515 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 03:01:48.839100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839099    5515 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 03:01:48.839217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839207    5515 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 03:01:48.839410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839398    5515 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 03:01:48.839443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839429    5515 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 03:01:48.839537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839528    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 03:01:48.839592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839543    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 03:01:48.839592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839572    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 03:01:48.839592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839579    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 03:01:48.839592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839585    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 03:01:48.839592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839591    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839596    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839602    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839608    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839614    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839623    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839632    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839656    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 03:01:48.839729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.839665    5515 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 03:01:48.843087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.843074    5515 server.go:1263] "Started kubelet"
Nov 03 03:01:48.843181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.843157    5515 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 03:01:48.843243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.843180    5515 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 03:01:48.843290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.843255    5515 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 03:01:48.843718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.843695    5515 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 03:01:48.843706 ip-10-0-138-11 systemd[1]: Started Kubernetes Kubelet.
Nov 03 03:01:48.846035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.845708    5515 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 03:01:48.847983 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.847966    5515 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.848017    5515 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.848146    5515 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 02:55:33 +0000 UTC" deadline="2028-07-18 12:29:33.277742812 +0000 UTC"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.848185    5515 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14961h27m44.429561921s"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.848257    5515 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.848266    5515 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.848341    5515 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:01:48.848887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.848369    5515 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 03:01:48.850478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.849873    5515 factory.go:55] Registering systemd factory
Nov 03 03:01:48.850478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.849892    5515 factory.go:223] Registration of the systemd container factory successfully
Nov 03 03:01:48.851842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.851821    5515 factory.go:153] Registering CRI-O factory
Nov 03 03:01:48.851989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.851846    5515 factory.go:223] Registration of the crio container factory successfully
Nov 03 03:01:48.851989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.851913    5515 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 03:01:48.851989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.851932    5515 factory.go:103] Registering Raw factory
Nov 03 03:01:48.851989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.851946    5515 manager.go:1199] Started watching for new ooms in manager
Nov 03 03:01:48.852278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.852266    5515 manager.go:319] Starting recovery of all containers
Nov 03 03:01:48.856577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856494    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc29a623-58c0-41a7-9167-926cc8e5b8c8" volumeName="kubernetes.io/empty-dir/bc29a623-58c0-41a7-9167-926cc8e5b8c8-tmp-dir" seLinuxMountContext=""
Nov 03 03:01:48.856577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856571    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856589    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856601    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856614    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856626    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44942777-b4b6-4264-aeff-7c88943cf14e" volumeName="kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-cni-binary-copy" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856637    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856648    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09cc1ed-086f-4b12-a9ff-e465fbe94038" volumeName="kubernetes.io/projected/f09cc1ed-086f-4b12-a9ff-e465fbe94038-kube-api-access-8d5vf" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856662    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856674    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856685    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81335b4a-183a-49d7-81d0-8bff20d94e77" volumeName="kubernetes.io/projected/81335b4a-183a-49d7-81d0-8bff20d94e77-kube-api-access-kx4f4" seLinuxMountContext=""
Nov 03 03:01:48.856700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856701    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" volumeName="kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856712    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856726    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856741    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="150b0a73-6cfc-438d-853f-3302f2232ec2" volumeName="kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856753    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="150b0a73-6cfc-438d-853f-3302f2232ec2" volumeName="kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856764    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856777    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" volumeName="kubernetes.io/projected/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-kube-api-access-rpls9" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856788    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09cc1ed-086f-4b12-a9ff-e465fbe94038" volumeName="kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856828    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="794ed6bd-5ce9-4aea-99ab-75607b2cd846" volumeName="kubernetes.io/projected/794ed6bd-5ce9-4aea-99ab-75607b2cd846-kube-api-access-rdb6f" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856843    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856855    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c603e8-5a0a-42af-a866-3439e1fdef56" volumeName="kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-env-overrides" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856868    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856879    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856890    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856903    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856919    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" volumeName="kubernetes.io/empty-dir/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-textfile" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856928    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c603e8-5a0a-42af-a866-3439e1fdef56" volumeName="kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856963    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35125e2e-9a2d-4577-98ca-5966cfe1d3a1" volumeName="kubernetes.io/configmap/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-konnectivity-ca" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856973    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="794ed6bd-5ce9-4aea-99ab-75607b2cd846" volumeName="kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-tuned" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856985    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c603e8-5a0a-42af-a866-3439e1fdef56" volumeName="kubernetes.io/projected/a3c603e8-5a0a-42af-a866-3439e1fdef56-kube-api-access-xhbdq" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.856997    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f023c708-c71a-4dcb-99cd-f604cb0088ce" volumeName="kubernetes.io/projected/f023c708-c71a-4dcb-99cd-f604cb0088ce-kube-api-access-2ptbp" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857005    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857012    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857020    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5000ce1d-93ee-4bed-bf15-7ecb2d1301e9" volumeName="kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857028    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857036    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" volumeName="kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-metrics-client-ca" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857043    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" volumeName="kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857052    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c603e8-5a0a-42af-a866-3439e1fdef56" volumeName="kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-config" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857059    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc29a623-58c0-41a7-9167-926cc8e5b8c8" volumeName="kubernetes.io/projected/bc29a623-58c0-41a7-9167-926cc8e5b8c8-kube-api-access-tfwr2" seLinuxMountContext=""
Nov 03 03:01:48.857075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857066    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857076    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09cc1ed-086f-4b12-a9ff-e465fbe94038" volumeName="kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-binary-copy" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857203    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857213    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857222    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996d2ead-9513-41c2-acd6-1ced1f624131" volumeName="kubernetes.io/projected/996d2ead-9513-41c2-acd6-1ced1f624131-kube-api-access-xdgk8" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857230    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff3b7b52-b0ad-4965-ba72-39330944d766" volumeName="kubernetes.io/projected/ff3b7b52-b0ad-4965-ba72-39330944d766-kube-api-access-v9kdd" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857237    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44942777-b4b6-4264-aeff-7c88943cf14e" volumeName="kubernetes.io/projected/44942777-b4b6-4264-aeff-7c88943cf14e-kube-api-access-7v9dm" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857248    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996d2ead-9513-41c2-acd6-1ced1f624131" volumeName="kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857260    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d44eda-e0e2-4bea-afeb-160eebdc1710" volumeName="kubernetes.io/configmap/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857271    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d44eda-e0e2-4bea-afeb-160eebdc1710" volumeName="kubernetes.io/empty-dir/c7d44eda-e0e2-4bea-afeb-160eebdc1710-data-volume" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857284    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d44eda-e0e2-4bea-afeb-160eebdc1710" volumeName="kubernetes.io/projected/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-api-access-dsp96" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857296    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857307    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857319    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f023c708-c71a-4dcb-99cd-f604cb0088ce" volumeName="kubernetes.io/secret/f023c708-c71a-4dcb-99cd-f604cb0088ce-cert" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857331    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="150b0a73-6cfc-438d-853f-3302f2232ec2" volumeName="kubernetes.io/projected/150b0a73-6cfc-438d-853f-3302f2232ec2-kube-api-access-6hcn5" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857342    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a2050bd-da88-4923-b686-4c9e70226b59" volumeName="kubernetes.io/configmap/7a2050bd-da88-4923-b686-4c9e70226b59-iptables-alerter-script" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857366    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857376    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7d44eda-e0e2-4bea-afeb-160eebdc1710" volumeName="kubernetes.io/secret/c7d44eda-e0e2-4bea-afeb-160eebdc1710-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857388    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/empty-dir/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-audit-log" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857400    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/projected/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-kube-api-access-6qlvb" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857410    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857422    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857434    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857445    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="794ed6bd-5ce9-4aea-99ab-75607b2cd846" volumeName="kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-tmp" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857457    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857470    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-client-ca-bundle" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857481    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09cc1ed-086f-4b12-a9ff-e465fbe94038" volumeName="kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857494    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35125e2e-9a2d-4577-98ca-5966cfe1d3a1" volumeName="kubernetes.io/secret/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-agent-certs" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857506    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64651a3d-f058-40c3-9171-53ba4e35d04a" volumeName="kubernetes.io/configmap/64651a3d-f058-40c3-9171-53ba4e35d04a-serviceca" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857517    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a2050bd-da88-4923-b686-4c9e70226b59" volumeName="kubernetes.io/projected/7a2050bd-da88-4923-b686-4c9e70226b59-kube-api-access-xxb44" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857529    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857541    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857570    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857580    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857597    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c603e8-5a0a-42af-a866-3439e1fdef56" volumeName="kubernetes.io/secret/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857609    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44942777-b4b6-4264-aeff-7c88943cf14e" volumeName="kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-multus-daemon-config" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857621    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63321407-0b4f-4f4c-9f17-19aaf6ef081a" volumeName="kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857633    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc29a623-58c0-41a7-9167-926cc8e5b8c8" volumeName="kubernetes.io/configmap/bc29a623-58c0-41a7-9167-926cc8e5b8c8-config-volume" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857644    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc29a623-58c0-41a7-9167-926cc8e5b8c8" volumeName="kubernetes.io/secret/bc29a623-58c0-41a7-9167-926cc8e5b8c8-metrics-tls" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857656    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857668    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcff55ad-f5a9-4f59-afd0-ecae4aefa472" volumeName="kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857679    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857691    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81335b4a-183a-49d7-81d0-8bff20d94e77" volumeName="kubernetes.io/empty-dir/81335b4a-183a-49d7-81d0-8bff20d94e77-tmp-dir" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857703    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857714    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" volumeName="kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857727    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" volumeName="kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-tls" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857740    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9d44588-4826-47de-bd89-fa91a3ed291d" volumeName="kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857753    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05f28420-10c0-44c8-a199-296d755fa0dc" volumeName="kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857765    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="150b0a73-6cfc-438d-853f-3302f2232ec2" volumeName="kubernetes.io/configmap/150b0a73-6cfc-438d-853f-3302f2232ec2-metrics-client-ca" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857776    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21108ceb-31ac-4158-bc6f-10553563603b" volumeName="kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857788    5515 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64651a3d-f058-40c3-9171-53ba4e35d04a" volumeName="kubernetes.io/projected/64651a3d-f058-40c3-9171-53ba4e35d04a-kube-api-access-5skpl" seLinuxMountContext=""
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857800    5515 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 03:01:48.858251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.857808    5515 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 03:01:48.884442 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.884391    5515 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 03:01:48.885913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.885890    5515 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 03:01:48.885913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.885916    5515 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 03:01:48.886098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.885943    5515 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 03:01:48.886098 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.885985    5515 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 03:01:48.895399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.895370    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:01:48.896599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.896566    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:01:48.898869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.898840    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:01:48.899519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.899496    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:01:48.900943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.900916    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:01:48.901387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.901260    5515 generic.go:356] "Generic (PLEG): container finished" podID="a3c603e8-5a0a-42af-a866-3439e1fdef56" containerID="c1a03c31d13ef4cff44a73b7727cc26c4b9279d71688f898346b09c164815a00" exitCode=1
Nov 03 03:01:48.910313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.910286    5515 generic.go:356] "Generic (PLEG): container finished" podID="9e897c78-ce55-4d3f-bcfd-148c8faf01f2" containerID="7a48b756d91375fccb796c297b10d915be6568847cdaaaea784b4057a06643d8" exitCode=0
Nov 03 03:01:48.920258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.920238    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:01:48.920381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.920268    5515 generic.go:356] "Generic (PLEG): container finished" podID="44942777-b4b6-4264-aeff-7c88943cf14e" containerID="ba986d3f84cc2ed9819164c8bb22ab5e7a5b61927c508b6e5472ad792469c6c3" exitCode=1
Nov 03 03:01:48.928826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.928802    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="2186c7bcb7ea7471b917d29cbc78d35807c011607877376302f77d6ec6ef55d5" exitCode=0
Nov 03 03:01:48.936455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.936422    5515 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="11ac97b2ce165a6b489f2e051f946ff7c72ff2d384419706ed9cc34b4229af1e" exitCode=0
Nov 03 03:01:48.936455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.936453    5515 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="b8aff79b4aef145e5e46719b51199f54ce10f19f29131685ec5eee4070d7ce79" exitCode=0
Nov 03 03:01:48.936455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.936461    5515 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="4e1a2b31125cda4b7252dc3e761c22698d3fa74d310ddbf22e0af0c53289ea4d" exitCode=0
Nov 03 03:01:48.936734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.936466    5515 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="f13fd4fdd566821d0db7dc4c59b36688eee4181a26d93ae9ec361bebcba28026" exitCode=0
Nov 03 03:01:48.936734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.936473    5515 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="792a9767c89ad81d2cad39797d861c1d062ece44a38187455fc276e0da72be6d" exitCode=0
Nov 03 03:01:48.936734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.936481    5515 generic.go:356] "Generic (PLEG): container finished" podID="f09cc1ed-086f-4b12-a9ff-e465fbe94038" containerID="af83ccf6dd21a71f70106fe25c5a64d6b7699d268cda896b4818ddfc2e502702" exitCode=0
Nov 03 03:01:48.940211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.940190    5515 generic.go:356] "Generic (PLEG): container finished" podID="f0a743d363639b28a6d7229f5e9636d9" containerID="adf936ee0d17157f6cd6ddd2e537f582de8702665abd5e339e745d472969820d" exitCode=0
Nov 03 03:01:48.948430 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.948402    5515 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:01:48.962150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.962078    5515 manager.go:324] Recovery completed
Nov 03 03:01:48.980423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.980404    5515 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:01:48.982676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.982657    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:01:48.982789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.982694    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:01:48.982789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.982707    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:01:48.985058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985039    5515 cpu_manager.go:226] "Starting" policy="none"
Nov 03 03:01:48.985058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985055    5515 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:01:48.985193 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985057    5515 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 03:01:48.985193 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985124    5515 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 03:01:48.985276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985262    5515 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 03:01:48.985307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985277    5515 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 03:01:48.985307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985294    5515 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 03:01:48.985307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985300    5515 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 03:01:48.985307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985306    5515 policy_none.go:50] "Start"
Nov 03 03:01:48.985464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985315    5515 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 03:01:48.985464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985324    5515 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 03:01:48.985464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985447    5515 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 03:01:48.985464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985454    5515 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 03:01:48.985464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985459    5515 policy_none.go:44] "Start"
Nov 03 03:01:48.985947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985932    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:01:48.985992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985963    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:01:48.985992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.985974    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:01:48.986055 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.986047    5515 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 03:01:48.990016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.989981    5515 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 03:01:48.992698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.992686    5515 manager.go:341] "Starting Device Plugin manager"
Nov 03 03:01:48.992760 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.992711    5515 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 03:01:48.992760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.992721    5515 server.go:87] "Starting device plugin registration server"
Nov 03 03:01:48.992970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.992958    5515 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 03:01:48.993021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.992969    5515 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 03:01:48.993101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.993078    5515 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 03:01:48.993232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.993191    5515 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 03:01:48.993283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.993238    5515 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 03:01:48.993283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.993245    5515 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 03:01:48.993372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.993296    5515 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T03:01:48.993226719Z","UUID":"c490e6c2-3624-415b-9275-ff95b9cccf12","Handler":null,"Name":"","Endpoint":""}
Nov 03 03:01:48.995188 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.995164    5515 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 03:01:48.995301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.995193    5515 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 03:01:48.995301 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:48.995213    5515 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-11.us-west-1.compute.internal\" not found"
Nov 03 03:01:48.995301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:48.995223    5515 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 03:01:49.049942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.049917    5515 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 03:01:49.051649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.051629    5515 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:01:49.051722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.051629    5515 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:01:49.055141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.055122    5515 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:01:49.094258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.094224    5515 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 03:01:49.095337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.095311    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 03:01:49.095447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.095343    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 03:01:49.095447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.095358    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 03:01:49.095447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.095405    5515 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.151582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.151526    5515 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 03:01:49.187099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187042    5515 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"]
Nov 03 03:01:49.187723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187688    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"8452f8e9cc16a665620f2f318ca698f0","Type":"ContainerStarted","Data":"7cd13ed9e71a55608bbdf2859323b205e4c650e9835f5c515f0ede1d94e06995"}
Nov 03 03:01:49.187765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187733    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"8452f8e9cc16a665620f2f318ca698f0","Type":"ContainerStarted","Data":"6ba3f8087608dd2e1e8188ce9ac66eaf086a0e6eead7fb3620cf0b4e5a36fffc"}
Nov 03 03:01:49.187821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187791    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"f0a743d363639b28a6d7229f5e9636d9","Type":"ContainerStarted","Data":"ad75cc82740e9ae1ddc6f9c8be0b0b793f0a40ffdb9a0f55b1f9fdd56d6becc2"}
Nov 03 03:01:49.187821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187803    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"f0a743d363639b28a6d7229f5e9636d9","Type":"ContainerDied","Data":"adf936ee0d17157f6cd6ddd2e537f582de8702665abd5e339e745d472969820d"}
Nov 03 03:01:49.187821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187815    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal" event={"ID":"f0a743d363639b28a6d7229f5e9636d9","Type":"ContainerStarted","Data":"3efc44bcff304956bc1f355ecd22edb387e8b5f53fa4f3aefd879e3d6617374b"}
Nov 03 03:01:49.187915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187846    5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.187963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.187849    5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.207882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.207858    5515 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.208010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.207925    5515 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.208476 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.208460    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeNotReady"
Nov 03 03:01:49.208527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.208491    5515 setters.go:546] "Node became not ready" node="ip-10-0-138-11.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T03:01:49Z","lastTransitionTime":"2026-11-03T03:01:49Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 03:01:49.251358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.251282    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8452f8e9cc16a665620f2f318ca698f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"8452f8e9cc16a665620f2f318ca698f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.251358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.251329    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.251358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.251347    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.254314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.254294    5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 03:01:49.254370 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:49.254346    5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.254909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.254896    5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 03:01:49.254944 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:49.254933    5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.271047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.271022    5515 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-11.us-west-1.compute.internal" is owned by different node (oldNodeID="c628e2ff-fe55-4dd1-983a-432920dfbb82", newNodeID=""), cleaning up...
Nov 03 03:01:49.352040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.352006    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.352203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.352045    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.352203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.352071    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8452f8e9cc16a665620f2f318ca698f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"8452f8e9cc16a665620f2f318ca698f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.352203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.352088    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.352203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.352109    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8452f8e9cc16a665620f2f318ca698f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"8452f8e9cc16a665620f2f318ca698f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.352203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.352159    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0a743d363639b28a6d7229f5e9636d9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" (UID: \"f0a743d363639b28a6d7229f5e9636d9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.581996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.581923    5515 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-11.us-west-1.compute.internal" event="NodeReady"
Nov 03 03:01:49.582125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.582006    5515 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 03:01:49.834745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.834663    5515 apiserver.go:52] "Watching apiserver"
Nov 03 03:01:49.950372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.950345    5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.950516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.950422    5515 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:49.984313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.984276    5515 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 03:01:49.986159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:49.986127    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rqm5c","openshift-ingress-canary/ingress-canary-mrlzv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-hgj4z","openshift-console/console-dfdd8856b-dgsz5","openshift-multus/multus-jb7sk","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb","openshift-ovn-kubernetes/ovnkube-node-cvkw5","openshift-cluster-node-tuning-operator/tuned-hgh8q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4","openshift-multus/multus-additional-cni-plugins-rb9rv","openshift-multus/network-metrics-daemon-6k7g2","kube-system/konnectivity-agent-5xflz","openshift-console/console-67c65cdb4b-65nnw","openshift-console/console-6776568cb9-jl5c5","kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal","openshift-dns/dns-default-8r26d","openshift-dns/node-resolver-77z26","openshift-network-diagnostics/network-check-target-568md","openshift-network-operator/iptables-alerter-gmpwj","openshift-monitoring/metrics-server-875cc949b-lpm9j","openshift-monitoring/node-exporter-wwq7p","openshift-image-registry/node-ca-4msjq"]
Nov 03 03:01:50.029337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.029308    5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 03:01:50.029488 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:50.029361    5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:50.034876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.034857    5515 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 03:01:50.034954 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:50.034903    5515 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-11.us-west-1.compute.internal"
Nov 03 03:01:50.050929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.050909    5515 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 03:01:50.054610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.054586    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.054730 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.054611    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.054811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.054789    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-node-log\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.054867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.054829    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.054905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.054863    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-sys-fs\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.054997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.054975    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.055077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055058    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.055133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055059    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.055181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055135    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-hostroot\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.055227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055137    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.055227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055191    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skpl\" (UniqueName: \"kubernetes.io/projected/64651a3d-f058-40c3-9171-53ba4e35d04a-kube-api-access-5skpl\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.055227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055219    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-metrics-client-ca\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055227    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055250    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055261    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055282    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055296    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-etc-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055315    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055345    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-bin\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055358    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055386    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.055407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055386    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055458    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81335b4a-183a-49d7-81d0-8bff20d94e77-hosts-file\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055498    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-client-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055541    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055563    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055609    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055665    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055679    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055687    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-conf\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055702    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055718    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055735    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-sys\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055768    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055790    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055788    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-kubelet\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.055864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055857    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-lib-modules\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055893    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hrjcl\""
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055987    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.055979    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056019    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcn5\" (UniqueName: \"kubernetes.io/projected/150b0a73-6cfc-438d-853f-3302f2232ec2-kube-api-access-6hcn5\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056047    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-kubelet\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056061    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-conf-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056088    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9dm\" (UniqueName: \"kubernetes.io/projected/44942777-b4b6-4264-aeff-7c88943cf14e-kube-api-access-7v9dm\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056129    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056146    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fg6rm\""
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056179    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056240    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056252    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.056443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056253    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.057503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057482    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tvk4\""
Nov 03 03:01:50.057625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056748    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rnqls\""
Nov 03 03:01:50.057625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056840    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d44eda-e0e2-4bea-afeb-160eebdc1710-crio-socket\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.057625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057598    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-dbus\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.057782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057646    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.057782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057659    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fp5b4\""
Nov 03 03:01:50.057782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057696    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.057782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057732    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.057782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057765    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-system-cni-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057806    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-tmp\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057846    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4f4\" (UniqueName: \"kubernetes.io/projected/81335b4a-183a-49d7-81d0-8bff20d94e77-kube-api-access-kx4f4\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057881    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057905    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zcpl5\""
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057913    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-audit-log\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057939    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-multus-certs\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057971    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.058001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057997    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058002    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-k8s-cni-cncf-io\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058033    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-etc-kubernetes\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058063    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptbp\" (UniqueName: \"kubernetes.io/projected/f023c708-c71a-4dcb-99cd-f604cb0088ce-kube-api-access-2ptbp\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058073    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wrqj7\""
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058090    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058121    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058152    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-log-socket\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058182    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqjzv\""
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058187    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058220    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-modprobe-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058246    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058276    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64651a3d-f058-40c3-9171-53ba4e35d04a-host\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058289    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 03:01:50.058340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058305    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-etc-selinux\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058356    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058398    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-netd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058429    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-kubernetes\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058457    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ckmp9\""
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058467    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-metrics-server-audit-profiles\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058495    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-var-lib-kubelet\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058526    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d44eda-e0e2-4bea-afeb-160eebdc1710-data-volume\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058572    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cgzd9\""
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058584    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058614    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-socket-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058642    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-registration-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058670    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbdq\" (UniqueName: \"kubernetes.io/projected/a3c603e8-5a0a-42af-a866-3439e1fdef56-kube-api-access-xhbdq\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058696    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058700    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-root\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058731    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-system-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058763    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-bin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058797    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058836    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-systemd-units\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.056906    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.058923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058889    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-sys\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.059769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058977    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.059769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.057736    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d2r77\""
Nov 03 03:01:50.059769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.059252    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-tmp\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.059769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.059322    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-original-pull-secret\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.059769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.059574    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7d44eda-e0e2-4bea-afeb-160eebdc1710-data-volume\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.060021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.058769    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.060021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.059893    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.060021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.059935    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-audit-log\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.060155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060045    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.060207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060171    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.060262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060211    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.060353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060285    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.060353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060321    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-ovn\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.060492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060356    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.060492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060388    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-cnibin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.060492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060473    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-multus-daemon-config\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060508    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060539    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-tuned\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060595    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060625    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kdd\" (UniqueName: \"kubernetes.io/projected/ff3b7b52-b0ad-4965-ba72-39330944d766-kube-api-access-v9kdd\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060655    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060687    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-env-overrides\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060719    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpls9\" (UniqueName: \"kubernetes.io/projected/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-kube-api-access-rpls9\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060895    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060931    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.060966    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-cni-binary-copy\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061000    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061035    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061065    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.061113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061104    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc29a623-58c0-41a7-9167-926cc8e5b8c8-config-volume\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.061886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061261    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.061886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061603    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-tuned\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.061886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061704    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-multus-daemon-config\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.062028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061914    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.062028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.061995    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.062112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062086    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-multus\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.062162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062096    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44942777-b4b6-4264-aeff-7c88943cf14e-cni-binary-copy\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.062162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062135    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc29a623-58c0-41a7-9167-926cc8e5b8c8-tmp-dir\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.062325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062305    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc29a623-58c0-41a7-9167-926cc8e5b8c8-tmp-dir\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.062716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062698    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-var-lib-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.062779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062740    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.062779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062769    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdb6f\" (UniqueName: \"kubernetes.io/projected/794ed6bd-5ce9-4aea-99ab-75607b2cd846-kube-api-access-rdb6f\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.062843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062804    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-textfile\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.062843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062830    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150b0a73-6cfc-438d-853f-3302f2232ec2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.062907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062896    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-textfile\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.062938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062905    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.062975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062949    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-run\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.063007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.062984    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.063051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063022    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-netns\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.063088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063066    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwr2\" (UniqueName: \"kubernetes.io/projected/bc29a623-58c0-41a7-9167-926cc8e5b8c8-kube-api-access-tfwr2\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.063120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063098    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5vf\" (UniqueName: \"kubernetes.io/projected/f09cc1ed-086f-4b12-a9ff-e465fbe94038-kube-api-access-8d5vf\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.063160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063121    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-netns\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.063198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063156    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-tls\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.063261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063241    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64651a3d-f058-40c3-9171-53ba4e35d04a-serviceca\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.063308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063295    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cnibin\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.063421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063404    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-config\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.063482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063461    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.063532 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063500    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-systemd\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.063594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063572    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.063594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063574    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64651a3d-f058-40c3-9171-53ba4e35d04a-serviceca\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.063682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063595    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.063682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063612    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.063682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063644    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-kubelet-config\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.063682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063660    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.063792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063701    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.063792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063742    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.063792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063773    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsp96\" (UniqueName: \"kubernetes.io/projected/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-api-access-dsp96\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.063878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063791    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:50.063878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063848    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-device-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.063878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063865    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-binary-copy\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.064008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063881    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f023c708-c71a-4dcb-99cd-f604cb0088ce-cert\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:50.064008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063896    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d44eda-e0e2-4bea-afeb-160eebdc1710-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.064008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063912    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovn-node-metrics-cert\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.064008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063963    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysconfig\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.064008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.063984    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-host\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064014    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064036    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064046    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996d2ead-9513-41c2-acd6-1ced1f624131-metrics-certs\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064057    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-client-certs\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064112    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgk8\" (UniqueName: \"kubernetes.io/projected/996d2ead-9513-41c2-acd6-1ced1f624131-kube-api-access-xdgk8\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064151    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064187    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-slash\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064205    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-binary-copy\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064207    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovn-node-metrics-cert\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064214    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-script-lib\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.064286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064273    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-wtmp\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064310    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-socket-dir-parent\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064356    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-agent-certs\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064403    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc29a623-58c0-41a7-9167-926cc8e5b8c8-metrics-tls\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064451    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-script-lib\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064466    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-tls\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064513    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxb44\" (UniqueName: \"kubernetes.io/projected/7a2050bd-da88-4923-b686-4c9e70226b59-kube-api-access-xxb44\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064560    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064590    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064619    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064646    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81335b4a-183a-49d7-81d0-8bff20d94e77-tmp-dir\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064687    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-agent-certs\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064698    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlvb\" (UniqueName: \"kubernetes.io/projected/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-kube-api-access-6qlvb\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064727    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81335b4a-183a-49d7-81d0-8bff20d94e77-tmp-dir\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064731    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a2050bd-da88-4923-b686-4c9e70226b59-iptables-alerter-script\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064761    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064777    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2050bd-da88-4923-b686-4c9e70226b59-host-slash\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.064796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064805    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.065318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064824    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-konnectivity-ca\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:01:50.065318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064900    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a2050bd-da88-4923-b686-4c9e70226b59-iptables-alerter-script\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.065318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064922    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-systemd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.065318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.064986    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35125e2e-9a2d-4577-98ca-5966cfe1d3a1-konnectivity-ca\") pod \"konnectivity-agent-5xflz\" (UID: \"35125e2e-9a2d-4577-98ca-5966cfe1d3a1\") " pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:01:50.134750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.134720    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgk8\" (UniqueName: \"kubernetes.io/projected/996d2ead-9513-41c2-acd6-1ced1f624131-kube-api-access-xdgk8\") pod \"network-metrics-daemon-6k7g2\" (UID: \"996d2ead-9513-41c2-acd6-1ced1f624131\") " pod="openshift-multus/network-metrics-daemon-6k7g2"
Nov 03 03:01:50.134931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.134913    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9dm\" (UniqueName: \"kubernetes.io/projected/44942777-b4b6-4264-aeff-7c88943cf14e-kube-api-access-7v9dm\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.134994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.134967    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5vf\" (UniqueName: \"kubernetes.io/projected/f09cc1ed-086f-4b12-a9ff-e465fbe94038-kube-api-access-8d5vf\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.143708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.143662    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skpl\" (UniqueName: \"kubernetes.io/projected/64651a3d-f058-40c3-9171-53ba4e35d04a-kube-api-access-5skpl\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.143708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.143685    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4f4\" (UniqueName: \"kubernetes.io/projected/81335b4a-183a-49d7-81d0-8bff20d94e77-kube-api-access-kx4f4\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.144034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.144018    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnz9k\" (UniqueName: \"kubernetes.io/projected/63321407-0b4f-4f4c-9f17-19aaf6ef081a-kube-api-access-bnz9k\") pod \"network-check-target-568md\" (UID: \"63321407-0b4f-4f4c-9f17-19aaf6ef081a\") " pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:50.154102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.154085    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdb6f\" (UniqueName: \"kubernetes.io/projected/794ed6bd-5ce9-4aea-99ab-75607b2cd846-kube-api-access-rdb6f\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.154663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.154647    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwr2\" (UniqueName: \"kubernetes.io/projected/bc29a623-58c0-41a7-9167-926cc8e5b8c8-kube-api-access-tfwr2\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.154707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.154670    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbdq\" (UniqueName: \"kubernetes.io/projected/a3c603e8-5a0a-42af-a866-3439e1fdef56-kube-api-access-xhbdq\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.164196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.164179    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxb44\" (UniqueName: \"kubernetes.io/projected/7a2050bd-da88-4923-b686-4c9e70226b59-kube-api-access-xxb44\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.165294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165277    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-netns\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.165336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165306    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-netns\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.165336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165325    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cnibin\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.165430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165341    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165377    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-netns\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.165430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165380    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-systemd\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165382    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-netns\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.165430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165415    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cnibin\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165429    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-systemd\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165442    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-kubelet-config\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165469    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165496    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165505    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-device-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165534    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysconfig\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165543    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-kubelet-config\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165573    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165566    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-host\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165605    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-host\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165608    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-device-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165627    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-slash\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165636    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysconfig\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.165665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165656    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-wtmp\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165686    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-socket-dir-parent\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165706    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-slash\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165722    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-wtmp\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165729    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-socket-dir-parent\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165724    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2050bd-da88-4923-b686-4c9e70226b59-host-slash\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165757    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165777    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-systemd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165757    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2050bd-da88-4923-b686-4c9e70226b59-host-slash\") pod \"iptables-alerter-gmpwj\" (UID: \"7a2050bd-da88-4923-b686-4c9e70226b59\") " pod="openshift-network-operator/iptables-alerter-gmpwj"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165806    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165804    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-node-log\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165852    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-sys-fs\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165870    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-hostroot\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165877    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-node-log\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165824    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-systemd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165886    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-etc-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165913    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-sys-fs\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165912    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-bin\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165927    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-hostroot\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165936    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81335b4a-183a-49d7-81d0-8bff20d94e77-hosts-file\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165941    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-etc-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165946    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-bin\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165953    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-conf\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165981    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-sys\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165996    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-kubelet\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166012    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-lib-modules\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166029    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-kubelet\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166042    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-sys\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166054    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-kubelet\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166046    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-conf-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.165981    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81335b4a-183a-49d7-81d0-8bff20d94e77-hosts-file\") pod \"node-resolver-77z26\" (UID: \"81335b4a-183a-49d7-81d0-8bff20d94e77\") " pod="openshift-dns/node-resolver-77z26"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166082    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-multus-conf-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166085    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-kubelet\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166108    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d44eda-e0e2-4bea-afeb-160eebdc1710-crio-socket\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166135    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-dbus\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166140    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-lib-modules\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166140    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c7d44eda-e0e2-4bea-afeb-160eebdc1710-crio-socket\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166158    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-sysctl-conf\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166162    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166195    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-system-cni-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166216    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-multus-certs\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.166199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166223    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5000ce1d-93ee-4bed-bf15-7ecb2d1301e9-dbus\") pod \"global-pull-secret-syncer-rqm5c\" (UID: \"5000ce1d-93ee-4bed-bf15-7ecb2d1301e9\") " pod="kube-system/global-pull-secret-syncer-rqm5c"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166240    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-k8s-cni-cncf-io\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166256    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-etc-kubernetes\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166259    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-system-cni-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166273    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-log-socket\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166274    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-multus-certs\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166274    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f09cc1ed-086f-4b12-a9ff-e465fbe94038-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166294    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166305    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-run-k8s-cni-cncf-io\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166313    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-log-socket\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166313    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-etc-kubernetes\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166314    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-modprobe-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166339    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166350    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64651a3d-f058-40c3-9171-53ba4e35d04a-host\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166369    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-etc-selinux\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166378    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-modprobe-d\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166387    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-netd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166402    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-kubernetes\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166411    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-etc-selinux\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166419    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-var-lib-kubelet\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166401    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64651a3d-f058-40c3-9171-53ba4e35d04a-host\") pod \"node-ca-4msjq\" (UID: \"64651a3d-f058-40c3-9171-53ba4e35d04a\") " pod="openshift-image-registry/node-ca-4msjq"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166433    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-cni-netd\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166455    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-var-lib-kubelet\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166454    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-socket-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166455    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-etc-kubernetes\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166484    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-registration-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166492    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-socket-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166501    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-root\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166538    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b7b52-b0ad-4965-ba72-39330944d766-registration-dir\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166517    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-system-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166584    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-bin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166606    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-systemd-units\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166612    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-system-cni-dir\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166620    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-sys\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166637    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-bin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166563    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-root\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166653    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-systemd-units\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166660    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-ovn\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166638    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-ovn\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166682    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-cnibin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166684    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-sys\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166715    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-cnibin\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166720    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166737    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-run-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166750    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-multus\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166779    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-var-lib-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166806    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166826    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44942777-b4b6-4264-aeff-7c88943cf14e-host-var-lib-cni-multus\") pod \"multus-jb7sk\" (UID: \"44942777-b4b6-4264-aeff-7c88943cf14e\") " pod="openshift-multus/multus-jb7sk"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166842    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-run\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166863    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-var-lib-openvswitch\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166866    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3c603e8-5a0a-42af-a866-3439e1fdef56-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.167523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.166913    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/794ed6bd-5ce9-4aea-99ab-75607b2cd846-run\") pod \"tuned-hgh8q\" (UID: \"794ed6bd-5ce9-4aea-99ab-75607b2cd846\") " pod="openshift-cluster-node-tuning-operator/tuned-hgh8q"
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167644    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167666    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167710    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167741    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167758    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167759    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167780    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167811    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167815    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167855    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.167910    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.168047    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.168053    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 03:01:50.169232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.168980    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.169928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.169909    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-metrics-server-audit-profiles\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.170028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.170014    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 03:01:50.170197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.170181    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.170425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.170407    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpls9\" (UniqueName: \"kubernetes.io/projected/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-kube-api-access-rpls9\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.171495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.171479    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.171596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.171577    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.171691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.171637    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-env-overrides\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.171823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.171806    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 03:01:50.171885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.171836    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 03:01:50.171885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.171855    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 03:01:50.172078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.172052    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.172118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.172065    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 03:01:50.172424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.172409    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 03:01:50.172424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.172415    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ccc7x\""
Nov 03 03:01:50.172518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.172426    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc29a623-58c0-41a7-9167-926cc8e5b8c8-config-volume\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.173894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.173878    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.174006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.173991    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.174050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.174025    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3c603e8-5a0a-42af-a866-3439e1fdef56-ovnkube-config\") pod \"ovnkube-node-cvkw5\" (UID: \"a3c603e8-5a0a-42af-a866-3439e1fdef56\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:50.174050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.174025    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.174961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.174945    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcn5\" (UniqueName: \"kubernetes.io/projected/150b0a73-6cfc-438d-853f-3302f2232ec2-kube-api-access-6hcn5\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.175095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.175082    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f09cc1ed-086f-4b12-a9ff-e465fbe94038-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rb9rv\" (UID: \"f09cc1ed-086f-4b12-a9ff-e465fbe94038\") " pod="openshift-multus/multus-additional-cni-plugins-rb9rv"
Nov 03 03:01:50.175437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.175421    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.175613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.175598    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.176740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.176726    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.176740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.176731    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.178146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.178127    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kdd\" (UniqueName: \"kubernetes.io/projected/ff3b7b52-b0ad-4965-ba72-39330944d766-kube-api-access-v9kdd\") pod \"aws-ebs-csi-driver-node-hhxf4\" (UID: \"ff3b7b52-b0ad-4965-ba72-39330944d766\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hhxf4"
Nov 03 03:01:50.178213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.178186    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlvb\" (UniqueName: \"kubernetes.io/projected/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-kube-api-access-6qlvb\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.180044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.180029    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.180237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.180224    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsp96\" (UniqueName: \"kubernetes.io/projected/c7d44eda-e0e2-4bea-afeb-160eebdc1710-kube-api-access-dsp96\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.182237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.182223    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.204949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.204916    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 03:01:50.205613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.205580    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.205613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.205597    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz9pb\""
Nov 03 03:01:50.205613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.205595    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 03:01:50.205796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.205585    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 03:01:50.205796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.205643    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 03:01:50.206680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.206662    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snr2z\""
Nov 03 03:01:50.206960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.206947    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x6qz9\""
Nov 03 03:01:50.214635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214613    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f023c708-c71a-4dcb-99cd-f604cb0088ce-cert\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:50.214703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214613    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-client-certs\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.214748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214733    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 03:01:50.214780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214751    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc29a623-58c0-41a7-9167-926cc8e5b8c8-metrics-tls\") pod \"dns-default-8r26d\" (UID: \"bc29a623-58c0-41a7-9167-926cc8e5b8c8\") " pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:50.214967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214950    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 03:01:50.215019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214959    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 03:01:50.215019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214983    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mrslv\""
Nov 03 03:01:50.215019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.214987    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 03:01:50.215019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.215005    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pw7w9\""
Nov 03 03:01:50.215952 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.215934    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 03:01:50.216023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.215937    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 03:01:50.216023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.215972    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v5wbd\""
Nov 03 03:01:50.216023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.215975    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 03:01:50.216023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.216011    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 03:01:50.216023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.216014    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.216190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.216082    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 03:01:50.218976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.218961    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-atn5vmneqg5d8\""
Nov 03 03:01:50.219070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219053    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.219124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219066    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 03:01:50.219124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219077    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 03:01:50.219124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219067    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 03:01:50.219124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219080    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 03:01:50.219124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219113    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 03:01:50.219124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219068    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 03:01:50.219345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219119    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 03:01:50.219345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219140    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 03:01:50.219345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219157    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 03:01:50.219345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219181    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8skg7\""
Nov 03 03:01:50.219345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219226    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pqj9f\""
Nov 03 03:01:50.219345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219244    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.219717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219702    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.219828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219815    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.219942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.219929    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.220228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.220213    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.221596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.221580    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.221924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.221905    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.222093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222068    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.222211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222170    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.222258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222207    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.222258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222232    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.222338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222255    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.222338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222245    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.222338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222257    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150b0a73-6cfc-438d-853f-3302f2232ec2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.222432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.222341    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") pod \"console-6776568cb9-jl5c5\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") " pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:50.223704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.223685    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150b0a73-6cfc-438d-853f-3302f2232ec2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj4mb\" (UID: \"150b0a73-6cfc-438d-853f-3302f2232ec2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj4mb"
Nov 03 03:01:50.223704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.223692    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-tls\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.223857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.223767    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.223938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.223920    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.224938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.224901    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.224938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.224925    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.224938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.224910    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c7d44eda-e0e2-4bea-afeb-160eebdc1710-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hgj4z\" (UID: \"c7d44eda-e0e2-4bea-afeb-160eebdc1710\") " pod="openshift-insights/insights-runtime-extractor-hgj4z"
Nov 03 03:01:50.225120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.224917    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.225120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.224961    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-secret-metrics-server-tls\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.225120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.225002    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") pod \"console-67c65cdb4b-65nnw\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") " pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:01:50.225120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.225021    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") pod \"console-dfdd8856b-dgsz5\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") " pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:50.226179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.226162    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e897c78-ce55-4d3f-bcfd-148c8faf01f2-metrics-client-ca\") pod \"node-exporter-wwq7p\" (UID: \"9e897c78-ce55-4d3f-bcfd-148c8faf01f2\") " pod="openshift-monitoring/node-exporter-wwq7p"
Nov 03 03:01:50.226254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.226185    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:01:50.226302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.226252    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dcff55ad-f5a9-4f59-afd0-ecae4aefa472-client-ca-bundle\") pod \"metrics-server-875cc949b-lpm9j\" (UID: \"dcff55ad-f5a9-4f59-afd0-ecae4aefa472\") " pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:50.227708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.227692    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptbp\" (UniqueName: \"kubernetes.io/projected/f023c708-c71a-4dcb-99cd-f604cb0088ce-kube-api-access-2ptbp\") pod \"ingress-canary-mrlzv\" (UID: \"f023c708-c71a-4dcb-99cd-f604cb0088ce\") " pod="openshift-ingress-canary/ingress-canary-mrlzv"
Nov 03 03:01:50.423061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.422985    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:50.610687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.610658    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dfdd8856b-dgsz5"]
Nov 03 03:01:50.849516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.849438    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:50.957864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:50.957832    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerStarted","Data":"c9473dd98255a0a83ec8e40b641f918230186c0201001a619c4148e6afbd4d7c"}
Nov 03 03:01:50.984028 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:01:50.983994    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:01:51.403139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:51.403106    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8r26d"
Nov 03 03:01:51.958082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:51.958054    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:01:52.679595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:52.679472    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:01:53.310020 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:53.309989    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:54.502321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:54.502288    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:54.624520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:54.624494    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:01:54.859087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:54.859052    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:55.281657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:55.281588    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:01:55.281811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:55.281797    5515 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 03:01:55.949402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:55.949376    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:56.219602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:56.219513    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:56.733484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:56.733451    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:56.737071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:56.737051    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:58.507585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:58.507515    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:01:59.048337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:59.048306    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:59.212199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:59.212167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:59.792435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:59.792401    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:01:59.949629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:01:59.949593    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:00.180807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:00.180775    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:00.338922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:00.338878    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:02.493300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:02.493272    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5xflz"
Nov 03 03:02:02.613072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:02.613041    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:02:08.512742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:08.512716    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:02:09.435347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:09.435316    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:09.584634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:09.584595    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:11.424175 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:11.424150    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvkw5"
Nov 03 03:02:13.997361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:13.997330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:14.146649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:14.146616    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:15.013012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.012977    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67c65cdb4b-65nnw" podUID="e9d44588-4826-47de-bd89-fa91a3ed291d" containerName="console" containerID="cri-o://648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788" gracePeriod=15
Nov 03 03:02:15.248514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.248492    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:02:15.368841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.368817    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.368973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.368850    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.368973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.368886    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.369042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369005    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.369093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369058    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.369132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369122    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.369181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369165    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.369260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369239    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:15.369317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369265    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:15.369317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369246    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") pod \"e9d44588-4826-47de-bd89-fa91a3ed291d\" (UID: \"e9d44588-4826-47de-bd89-fa91a3ed291d\") "
Nov 03 03:02:15.369438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369335    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:02:15.369637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369544    5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9d44588-4826-47de-bd89-fa91a3ed291d-tmp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.369637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369592    5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.369637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369613    5515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-trusted-ca-bundle\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.370042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369661    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:15.370042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.369764    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:15.371634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.371610    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "kube-api-access-b4wcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:02:15.372183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.372129    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:15.372183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.372157    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config" pod "e9d44588-4826-47de-bd89-fa91a3ed291d" (UID: "e9d44588-4826-47de-bd89-fa91a3ed291d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:15.470174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.470150    5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-oauth-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.470174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.470170    5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9d44588-4826-47de-bd89-fa91a3ed291d-console-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.470300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.470206    5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-oauth-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.470300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.470219    5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9d44588-4826-47de-bd89-fa91a3ed291d-console-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.470300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.470228    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4wcj\" (UniqueName: \"kubernetes.io/projected/e9d44588-4826-47de-bd89-fa91a3ed291d-kube-api-access-b4wcj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:15.977490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:15.977433    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-dfdd8856b-dgsz5" podUID="05f28420-10c0-44c8-a199-296d755fa0dc" containerName="console" containerID="cri-o://dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1" gracePeriod=15
Nov 03 03:02:16.041145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.041110    5515 generic.go:356] "Generic (PLEG): container finished" podID="e9d44588-4826-47de-bd89-fa91a3ed291d" containerID="648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788" exitCode=0
Nov 03 03:02:16.041577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.041163    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c65cdb4b-65nnw"
Nov 03 03:02:16.041577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.041178    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c65cdb4b-65nnw" event={"ID":"e9d44588-4826-47de-bd89-fa91a3ed291d","Type":"ContainerDied","Data":"648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788"}
Nov 03 03:02:16.041577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.041232    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c65cdb4b-65nnw" event={"ID":"e9d44588-4826-47de-bd89-fa91a3ed291d","Type":"ContainerDied","Data":"ff71657a72c1533141609294f4497411da7f09a7788e92b641d0a8fd81f9ad80"}
Nov 03 03:02:16.041577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.041261    5515 scope.go:122] "RemoveContainer" containerID="648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788"
Nov 03 03:02:16.101020 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.100999    5515 scope.go:122] "RemoveContainer" containerID="648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788"
Nov 03 03:02:16.101333 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:02:16.101311    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788\": container with ID starting with 648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788 not found: ID does not exist" containerID="648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788"
Nov 03 03:02:16.101379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.101344    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788"} err="failed to get container status \"648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788\": rpc error: code = NotFound desc = could not find container \"648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788\": container with ID starting with 648c923db909518bcd8b615016607941c56fa5c5066a2256be47614414ec5788 not found: ID does not exist"
Nov 03 03:02:16.219923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.219892    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c65cdb4b-65nnw"]
Nov 03 03:02:16.223869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.223849    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67c65cdb4b-65nnw"]
Nov 03 03:02:16.263634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.263609    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:02:16.377846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.377815    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.377871    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.377904    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.377938    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.377983    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378022    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378068    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") pod \"05f28420-10c0-44c8-a199-296d755fa0dc\" (UID: \"05f28420-10c0-44c8-a199-296d755fa0dc\") "
Nov 03 03:02:16.378303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378105    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:02:16.378303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378284    5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f28420-10c0-44c8-a199-296d755fa0dc-tmp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.378502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378366    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:16.378577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378522    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:16.378577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.378527    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:16.380435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.380410    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "kube-api-access-9sl9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:02:16.380435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.380414    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:16.380594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.380476    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert" pod "05f28420-10c0-44c8-a199-296d755fa0dc" (UID: "05f28420-10c0-44c8-a199-296d755fa0dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:16.410789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.410757    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:16.478955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.478917    5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-oauth-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.478955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.478950    5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.478955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.478960    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sl9g\" (UniqueName: \"kubernetes.io/projected/05f28420-10c0-44c8-a199-296d755fa0dc-kube-api-access-9sl9g\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.479413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.478971    5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05f28420-10c0-44c8-a199-296d755fa0dc-console-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.479413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.478980    5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-oauth-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.479413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.478988    5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05f28420-10c0-44c8-a199-296d755fa0dc-console-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:02:16.586512 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.586441    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:16.747382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.747350    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:16.891379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:16.891340    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d44588-4826-47de-bd89-fa91a3ed291d" path="/var/lib/kubelet/pods/e9d44588-4826-47de-bd89-fa91a3ed291d/volumes"
Nov 03 03:02:17.047850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.047821    5515 generic.go:356] "Generic (PLEG): container finished" podID="05f28420-10c0-44c8-a199-296d755fa0dc" containerID="dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1" exitCode=0
Nov 03 03:02:17.048364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.047876    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfdd8856b-dgsz5"
Nov 03 03:02:17.048364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.047876    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfdd8856b-dgsz5" event={"ID":"05f28420-10c0-44c8-a199-296d755fa0dc","Type":"ContainerDied","Data":"dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1"}
Nov 03 03:02:17.048364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.047987    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfdd8856b-dgsz5" event={"ID":"05f28420-10c0-44c8-a199-296d755fa0dc","Type":"ContainerDied","Data":"ef70771c705eca2f9ac1c04a8a501db39c5699542809c03ae3d362ece38f338f"}
Nov 03 03:02:17.048364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.048017    5515 scope.go:122] "RemoveContainer" containerID="dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1"
Nov 03 03:02:17.055758 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.055728    5515 scope.go:122] "RemoveContainer" containerID="dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1"
Nov 03 03:02:17.056025 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:02:17.056007    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1\": container with ID starting with dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1 not found: ID does not exist" containerID="dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1"
Nov 03 03:02:17.056069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.056035    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1"} err="failed to get container status \"dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1\": rpc error: code = NotFound desc = could not find container \"dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1\": container with ID starting with dffe406791844aac79adf8a6a7a71e2699aee1a14b2c34a681942b27dfc16ca1 not found: ID does not exist"
Nov 03 03:02:17.226459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.226385    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dfdd8856b-dgsz5"]
Nov 03 03:02:17.276959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:17.276931    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dfdd8856b-dgsz5"]
Nov 03 03:02:18.891191 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:18.891160    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f28420-10c0-44c8-a199-296d755fa0dc" path="/var/lib/kubelet/pods/05f28420-10c0-44c8-a199-296d755fa0dc/volumes"
Nov 03 03:02:20.983223 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:02:20.983182    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:02:22.618394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:22.618361    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:02:22.622155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:22.622137    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-875cc949b-lpm9j"
Nov 03 03:02:22.682450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:22.682426    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-568md"
Nov 03 03:02:28.992534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:28.992501    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:29.138690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:29.138650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:47.683261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:47.683207    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:50.982308 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:02:50.982258    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:02:51.807020 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:51.806937    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:52.147799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:52.147732    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:02:54.088557 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088517    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:02:54.088955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088822    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="alertmanager" containerID="cri-o://e22567b2d9dc32fc9530ed14219acbb3e80ebd054e11e5fa762bb6b93617d455" gracePeriod=120
Nov 03 03:02:54.088955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088899    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-metric" containerID="cri-o://03fdc0ca8e31d21afe5b3c49458f95638f1b5f283355d292ff123cb11a5324da" gracePeriod=120
Nov 03 03:02:54.088955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088925    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="config-reloader" containerID="cri-o://5c7e0795e9abdf01874a6f9577b6fbcc52d85bfdec919a2e40b2bf4124a749cb" gracePeriod=120
Nov 03 03:02:54.089114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088944    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy" containerID="cri-o://cc8a8b6721db0a52bb5f5a8904e9b91ed00393b312fce35f98031512a3d5d4b3" gracePeriod=120
Nov 03 03:02:54.089114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088988    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="prom-label-proxy" containerID="cri-o://c9473dd98255a0a83ec8e40b641f918230186c0201001a619c4148e6afbd4d7c" gracePeriod=120
Nov 03 03:02:54.089114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:54.088905    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-web" containerID="cri-o://1e9c519f23c24214c9fd6967289cdca1a56fc323058c45286ff16b85a4cdf21b" gracePeriod=120
Nov 03 03:02:55.175426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175392    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="c9473dd98255a0a83ec8e40b641f918230186c0201001a619c4148e6afbd4d7c" exitCode=0
Nov 03 03:02:55.175426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175418    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="03fdc0ca8e31d21afe5b3c49458f95638f1b5f283355d292ff123cb11a5324da" exitCode=0
Nov 03 03:02:55.175426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175427    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="cc8a8b6721db0a52bb5f5a8904e9b91ed00393b312fce35f98031512a3d5d4b3" exitCode=0
Nov 03 03:02:55.175426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175432    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="1e9c519f23c24214c9fd6967289cdca1a56fc323058c45286ff16b85a4cdf21b" exitCode=0
Nov 03 03:02:55.175426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175437    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="5c7e0795e9abdf01874a6f9577b6fbcc52d85bfdec919a2e40b2bf4124a749cb" exitCode=0
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175444    5515 generic.go:356] "Generic (PLEG): container finished" podID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerID="e22567b2d9dc32fc9530ed14219acbb3e80ebd054e11e5fa762bb6b93617d455" exitCode=0
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175465    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"c9473dd98255a0a83ec8e40b641f918230186c0201001a619c4148e6afbd4d7c"}
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175528    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"03fdc0ca8e31d21afe5b3c49458f95638f1b5f283355d292ff123cb11a5324da"}
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175570    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"cc8a8b6721db0a52bb5f5a8904e9b91ed00393b312fce35f98031512a3d5d4b3"}
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175594    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"1e9c519f23c24214c9fd6967289cdca1a56fc323058c45286ff16b85a4cdf21b"}
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175617    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"5c7e0795e9abdf01874a6f9577b6fbcc52d85bfdec919a2e40b2bf4124a749cb"}
Nov 03 03:02:55.175944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:55.175639    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"e22567b2d9dc32fc9530ed14219acbb3e80ebd054e11e5fa762bb6b93617d455"}
Nov 03 03:02:57.761853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.761821    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-766f4dd556-x6pjk"]
Nov 03 03:02:57.762255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.762054    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e9d44588-4826-47de-bd89-fa91a3ed291d" containerName="console"
Nov 03 03:02:57.762255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.762063    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d44588-4826-47de-bd89-fa91a3ed291d" containerName="console"
Nov 03 03:02:57.762255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.762085    5515 cpu_manager.go:414] "containerMap: removing container" podUID="05f28420-10c0-44c8-a199-296d755fa0dc" containerName="console"
Nov 03 03:02:57.762255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.762090    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f28420-10c0-44c8-a199-296d755fa0dc" containerName="console"
Nov 03 03:02:57.762255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.762128    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d44588-4826-47de-bd89-fa91a3ed291d" containerName="console"
Nov 03 03:02:57.762255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.762137    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f28420-10c0-44c8-a199-296d755fa0dc" containerName="console"
Nov 03 03:02:57.763990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.763975    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.818040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.817999    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 03:02:57.818187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.818054    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 03:02:57.818187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.818088    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 03:02:57.868763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.868711    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 03:02:57.868763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.868752    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cfxsx\""
Nov 03 03:02:57.868763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.868767    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 03:02:57.876610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876587    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876619    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876649    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-serving-certs-ca-bundle\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876709    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-federate-client-tls\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876786    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-secret-telemeter-client\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876811    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-metrics-client-ca\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876883    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8v2t\" (UniqueName: \"kubernetes.io/projected/e5baba82-91af-4032-832a-2c12ac90ae0c-kube-api-access-h8v2t\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.876924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.876904    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-telemeter-client-tls\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.920362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.920333    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 03:02:57.921108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.921092    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766f4dd556-x6pjk"]
Nov 03 03:02:57.977830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.977800    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.977972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.977848    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-serving-certs-ca-bundle\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.977972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.977875    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-federate-client-tls\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.977972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.977914    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-secret-telemeter-client\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.977972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.977943    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-metrics-client-ca\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.978164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.977979    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8v2t\" (UniqueName: \"kubernetes.io/projected/e5baba82-91af-4032-832a-2c12ac90ae0c-kube-api-access-h8v2t\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.978164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.978004    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-telemeter-client-tls\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.978164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.978058    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.978334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.978264    5515 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 03:02:57.978696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.978640    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-serving-certs-ca-bundle\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.978978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.978965    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-metrics-client-ca\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.979139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.979115    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5baba82-91af-4032-832a-2c12ac90ae0c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.980715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.980697    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-telemeter-client-tls\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.980811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.980754    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.981073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.981057    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-federate-client-tls\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:57.981171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:57.981141    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e5baba82-91af-4032-832a-2c12ac90ae0c-secret-telemeter-client\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:58.042285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:58.042221    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8v2t\" (UniqueName: \"kubernetes.io/projected/e5baba82-91af-4032-832a-2c12ac90ae0c-kube-api-access-h8v2t\") pod \"telemeter-client-766f4dd556-x6pjk\" (UID: \"e5baba82-91af-4032-832a-2c12ac90ae0c\") " pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:58.075166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:58.075140    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk"
Nov 03 03:02:58.499474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:58.499451    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766f4dd556-x6pjk"]
Nov 03 03:02:58.501987 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:02:58.501958    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5baba82_91af_4032_832a_2c12ac90ae0c.slice/crio-fedf396f3fcd182ddf9b1d4ecd879ffaf4bed069b745902ec0a76cee6c0990a3 WatchSource:0}: Error finding container fedf396f3fcd182ddf9b1d4ecd879ffaf4bed069b745902ec0a76cee6c0990a3: Status 404 returned error can't find the container with id fedf396f3fcd182ddf9b1d4ecd879ffaf4bed069b745902ec0a76cee6c0990a3
Nov 03 03:02:58.503920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:58.503905    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:02:59.189008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:02:59.188969    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk" event={"ID":"e5baba82-91af-4032-832a-2c12ac90ae0c","Type":"ContainerStarted","Data":"fedf396f3fcd182ddf9b1d4ecd879ffaf4bed069b745902ec0a76cee6c0990a3"}
Nov 03 03:03:01.196916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.196825    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk" event={"ID":"e5baba82-91af-4032-832a-2c12ac90ae0c","Type":"ContainerStarted","Data":"be55ac59b0a3bb832045eee8a20f304a12d71213dacb230613c9ec7101619716"}
Nov 03 03:03:01.196916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.196886    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk" event={"ID":"e5baba82-91af-4032-832a-2c12ac90ae0c","Type":"ContainerStarted","Data":"8b7648c9b5a481e6b0f236aa0940f3a55b6f3dcbec4a5a3b3fdde1ad485305e2"}
Nov 03 03:03:01.196916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.196906    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk" event={"ID":"e5baba82-91af-4032-832a-2c12ac90ae0c","Type":"ContainerStarted","Data":"b237d6334c7fb246ae6a4c168d76fd9be6be9d985ef8ce3026a5147cc853c53c"}
Nov 03 03:03:01.322623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.322569    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-766f4dd556-x6pjk" podStartSLOduration=1.888569275 podStartE2EDuration="4.322540516s" totalImagesPullingTime="2.433971241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:03:01.32204915 +0000 UTC m=+72.552941568" watchObservedRunningTime="2026-11-03 03:03:01.322540516 +0000 UTC m=+72.553432927"
Nov 03 03:03:01.826498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.826466    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59f946c5cf-rf2lw"]
Nov 03 03:03:01.828492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.828475    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.890393 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.890367    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f946c5cf-rf2lw"]
Nov 03 03:03:01.907231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907209    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907239    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907255    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbhf\" (UniqueName: \"kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907274    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907353    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907411    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907430    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:01.907508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:01.907452    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.007931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.007894    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.007941    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.007960    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.007976    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbhf\" (UniqueName: \"kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.007994    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008017    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008050    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008068    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008398    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008801    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.008844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008819    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.009018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.008995    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.009071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.009005    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.010656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.010639    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.010814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.010794    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.070744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.070723    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbhf\" (UniqueName: \"kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf\") pod \"console-59f946c5cf-rf2lw\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") " pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.109076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.109012    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:03:02.138600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.138570    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:02.256477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.256448    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:03:02.513432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.513355    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f946c5cf-rf2lw"]
Nov 03 03:03:02.516289 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:03:02.516260    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47847104_e526_42fa_b564_61b6950d0562.slice/crio-fe0409c8f83b13ac33a79619862fd37950743d07da716d7b759f7505a7705aa3 WatchSource:0}: Error finding container fe0409c8f83b13ac33a79619862fd37950743d07da716d7b759f7505a7705aa3: Status 404 returned error can't find the container with id fe0409c8f83b13ac33a79619862fd37950743d07da716d7b759f7505a7705aa3
Nov 03 03:03:02.545733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.545703    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:03:02.727261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:02.727232    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:03:03.205993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:03.205953    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f946c5cf-rf2lw" event={"ID":"47847104-e526-42fa-b564-61b6950d0562","Type":"ContainerStarted","Data":"4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969"}
Nov 03 03:03:03.206165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:03.206002    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f946c5cf-rf2lw" event={"ID":"47847104-e526-42fa-b564-61b6950d0562","Type":"ContainerStarted","Data":"fe0409c8f83b13ac33a79619862fd37950743d07da716d7b759f7505a7705aa3"}
Nov 03 03:03:03.327600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:03.327530    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59f946c5cf-rf2lw" podStartSLOduration=2.327516771 podStartE2EDuration="2.327516771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:03:03.327013297 +0000 UTC m=+74.557905715" watchObservedRunningTime="2026-11-03 03:03:03.327516771 +0000 UTC m=+74.558409189"
Nov 03 03:03:12.139662 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:12.139626    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:12.140084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:12.139701    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:12.144418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:12.144402    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:12.240121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:12.240094    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:03:12.437809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:12.437695    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6776568cb9-jl5c5"]
Nov 03 03:03:20.982882 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:03:20.982846    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:03:37.457491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.457430    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6776568cb9-jl5c5" podUID="21108ceb-31ac-4158-bc6f-10553563603b" containerName="console" containerID="cri-o://c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058" gracePeriod=15
Nov 03 03:03:37.712068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.711999    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:03:37.780679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780641    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.780784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780689    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.780784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780760    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.780878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780801    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.780878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780828    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.780878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780859    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.781035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780883    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.781035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.780917    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") pod \"21108ceb-31ac-4158-bc6f-10553563603b\" (UID: \"21108ceb-31ac-4158-bc6f-10553563603b\") "
Nov 03 03:03:37.781193 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.781142    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:37.781409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.781209    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:37.781409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.781301    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:03:37.781521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.781430    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:37.781521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.781455    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:37.783341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.783316    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:03:37.783658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.783636    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "kube-api-access-fjsrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:03:37.783741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.783719    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config" pod "21108ceb-31ac-4158-bc6f-10553563603b" (UID: "21108ceb-31ac-4158-bc6f-10553563603b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:03:37.881991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.881953    5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.881991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.881984    5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-oauth-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.881991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.881993    5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-console-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.881991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.882002    5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21108ceb-31ac-4158-bc6f-10553563603b-tmp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.882257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.882012    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjsrx\" (UniqueName: \"kubernetes.io/projected/21108ceb-31ac-4158-bc6f-10553563603b-kube-api-access-fjsrx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.882257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.882024    5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21108ceb-31ac-4158-bc6f-10553563603b-console-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.882257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.882033    5515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-trusted-ca-bundle\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:37.882257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:37.882041    5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21108ceb-31ac-4158-bc6f-10553563603b-oauth-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:03:38.323000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.322967    5515 generic.go:356] "Generic (PLEG): container finished" podID="21108ceb-31ac-4158-bc6f-10553563603b" containerID="c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058" exitCode=0
Nov 03 03:03:38.323176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.323016    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6776568cb9-jl5c5"
Nov 03 03:03:38.323176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.323046    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6776568cb9-jl5c5" event={"ID":"21108ceb-31ac-4158-bc6f-10553563603b","Type":"ContainerDied","Data":"c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058"}
Nov 03 03:03:38.323176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.323085    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6776568cb9-jl5c5" event={"ID":"21108ceb-31ac-4158-bc6f-10553563603b","Type":"ContainerDied","Data":"a836549baf38978970430a777e5e4382b5a7166a0e47f7fec1d9a0750b8b0016"}
Nov 03 03:03:38.323176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.323104    5515 scope.go:122] "RemoveContainer" containerID="c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058"
Nov 03 03:03:38.332118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.332100    5515 scope.go:122] "RemoveContainer" containerID="c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058"
Nov 03 03:03:38.332401 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:03:38.332384    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058\": container with ID starting with c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058 not found: ID does not exist" containerID="c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058"
Nov 03 03:03:38.332448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.332411    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058"} err="failed to get container status \"c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058\": rpc error: code = NotFound desc = could not find container \"c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058\": container with ID starting with c4f6355ad194d49c029e9abf62095b4486d51070cc5553daf8cfd86816953058 not found: ID does not exist"
Nov 03 03:03:38.491490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.491453    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6776568cb9-jl5c5"]
Nov 03 03:03:38.542098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.542056    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6776568cb9-jl5c5"]
Nov 03 03:03:38.890847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:38.890818    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21108ceb-31ac-4158-bc6f-10553563603b" path="/var/lib/kubelet/pods/21108ceb-31ac-4158-bc6f-10553563603b/volumes"
Nov 03 03:03:47.988216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:03:47.988175    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:03:50.982868 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:03:50.982828    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:04:01.947424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:01.947383    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:04:20.982866 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:04:20.982829    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:04:48.325248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:48.325214    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:04:50.982521 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:04:50.982484    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:04:54.207295 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:04:54.207250    5515 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" podSandboxID="64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc"
Nov 03 03:04:54.207733 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:04:54.207305    5515 kuberuntime_manager.go:1939] "Failed to stop sandbox" podSandboxID={"Type":"cri-o","ID":"64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc"}
Nov 03 03:04:54.207733 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:04:54.207369    5515 kubelet.go:2243] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"" logger="UnhandledError"
Nov 03 03:04:54.207733 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:04:54.207384    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"" pod="openshift-monitoring/alertmanager-main-0" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"
Nov 03 03:04:56.135763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.135741    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.249767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249735    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.249942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249790    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.249942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249842    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.249942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249881    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.249942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249909    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.249942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249937    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.249969    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250002    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250032    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250066    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250116    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250146    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250180    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") pod \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\" (UID: \"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f\") "
Nov 03 03:04:56.250883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.250850    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:04:56.251474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.251239    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:04:56.253405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.253384    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:04:56.253586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.253510    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.254603 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.253736    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:04:56.254603 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.253758    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:04:56.254726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.253854    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.254726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.254025    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "kube-api-access-knr78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:04:56.254726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.254028    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.255006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.254982    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.255438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.255420    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.257798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.257736    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.264298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.264274    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config" pod "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" (UID: "8df4a68a-fd3f-4a36-9dda-f3b141bdec3f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:04:56.351598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351568    5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351593    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-metrics-client-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351604    5515 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-web-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351613    5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351623    5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351633    5515 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-main-db\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351641    5515 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-tls-assets\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351649    5515 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351657    5515 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-cluster-tls-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351664    5515 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-config-out\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351673    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knr78\" (UniqueName: \"kubernetes.io/projected/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-kube-api-access-knr78\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351681    5515 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-secret-alertmanager-main-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.351818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.351690    5515 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:04:56.575075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.574997    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8df4a68a-fd3f-4a36-9dda-f3b141bdec3f","Type":"ContainerDied","Data":"64cb7b26e91109fb3217f8399925ab9ea254359b938ca51e466891978b7e6ffc"}
Nov 03 03:04:56.575075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.575015    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.575075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.575069    5515 scope.go:122] "RemoveContainer" containerID="c9473dd98255a0a83ec8e40b641f918230186c0201001a619c4148e6afbd4d7c"
Nov 03 03:04:56.582352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.582332    5515 scope.go:122] "RemoveContainer" containerID="03fdc0ca8e31d21afe5b3c49458f95638f1b5f283355d292ff123cb11a5324da"
Nov 03 03:04:56.589051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.589033    5515 scope.go:122] "RemoveContainer" containerID="cc8a8b6721db0a52bb5f5a8904e9b91ed00393b312fce35f98031512a3d5d4b3"
Nov 03 03:04:56.595621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.595605    5515 scope.go:122] "RemoveContainer" containerID="1e9c519f23c24214c9fd6967289cdca1a56fc323058c45286ff16b85a4cdf21b"
Nov 03 03:04:56.601750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.601736    5515 scope.go:122] "RemoveContainer" containerID="5c7e0795e9abdf01874a6f9577b6fbcc52d85bfdec919a2e40b2bf4124a749cb"
Nov 03 03:04:56.608165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.608147    5515 scope.go:122] "RemoveContainer" containerID="e22567b2d9dc32fc9530ed14219acbb3e80ebd054e11e5fa762bb6b93617d455"
Nov 03 03:04:56.614615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.614599    5515 scope.go:122] "RemoveContainer" containerID="2186c7bcb7ea7471b917d29cbc78d35807c011607877376302f77d6ec6ef55d5"
Nov 03 03:04:56.745742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.745716    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:04:56.750481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.750460    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:04:56.796787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.796763    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797208    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="prom-label-proxy"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797229    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="prom-label-proxy"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797241    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-metric"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797247    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-metric"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797266    5515 cpu_manager.go:414] "containerMap: removing container" podUID="21108ceb-31ac-4158-bc6f-10553563603b" containerName="console"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797271    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21108ceb-31ac-4158-bc6f-10553563603b" containerName="console"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797283    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797287    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797302    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="init-config-reloader"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797310    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="init-config-reloader"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797327    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="config-reloader"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797331    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="config-reloader"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797344    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-web"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797349    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-web"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797360    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="alertmanager"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797366    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="alertmanager"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797443    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-metric"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797459    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="21108ceb-31ac-4158-bc6f-10553563603b" containerName="console"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797468    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797478    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="config-reloader"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797485    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="kube-rbac-proxy-web"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797494    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="alertmanager"
Nov 03 03:04:56.799719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.797500    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" containerName="prom-label-proxy"
Nov 03 03:04:56.803768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.803749    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.857959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.857932    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 03:04:56.858098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.857940    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 03:04:56.858098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.857949    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 03:04:56.858098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.857949    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 03:04:56.858098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.857980    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 03:04:56.858098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.858078    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 03:04:56.858327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.858095    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 03:04:56.858327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.858214    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x6qz9\""
Nov 03 03:04:56.858327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.858265    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 03:04:56.858704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.858677    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 03:04:56.890508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.890486    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df4a68a-fd3f-4a36-9dda-f3b141bdec3f" path="/var/lib/kubelet/pods/8df4a68a-fd3f-4a36-9dda-f3b141bdec3f/volumes"
Nov 03 03:04:56.917194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.917165    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:04:56.955579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955531    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955582    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955655    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955758 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955696    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-web-config\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955791    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955827    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955845    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq46z\" (UniqueName: \"kubernetes.io/projected/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-kube-api-access-zq46z\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955870    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955923    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.955977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.955965    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.956026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.956015    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.956076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.956056    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-config-volume\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:56.956113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:56.956097    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-config-out\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057143    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057178    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057200    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-web-config\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057316 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057243    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057381    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057414    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq46z\" (UniqueName: \"kubernetes.io/projected/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-kube-api-access-zq46z\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057505 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057456    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057505 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057488    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057532    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057601    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057615    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057630    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-config-volume\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057672    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-config-out\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.057870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.057701    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.058401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.058375    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.058566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.058530    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.060692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.060538    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.060871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.060835    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.060871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.060858    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-web-config\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.060997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.060863    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-config-volume\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.061313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.061288    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.061458 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.061437    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.061533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.061520    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-config-out\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.061982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.061963    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.062291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.062275    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.120252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.120197    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq46z\" (UniqueName: \"kubernetes.io/projected/860635a9-c37a-4f31-a4c3-b7d2a4ed5a20-kube-api-access-zq46z\") pod \"alertmanager-main-0\" (UID: \"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.126849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.126827    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 03:04:57.561776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.561704    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 03:04:57.565120 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:04:57.565092    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860635a9_c37a_4f31_a4c3_b7d2a4ed5a20.slice/crio-66ab27117686f82172085a7090b3be63dba2cb6485420a3e96ceba6ce2d2051e WatchSource:0}: Error finding container 66ab27117686f82172085a7090b3be63dba2cb6485420a3e96ceba6ce2d2051e: Status 404 returned error can't find the container with id 66ab27117686f82172085a7090b3be63dba2cb6485420a3e96ceba6ce2d2051e
Nov 03 03:04:57.583168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:57.583146    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"66ab27117686f82172085a7090b3be63dba2cb6485420a3e96ceba6ce2d2051e"}
Nov 03 03:04:58.588111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:58.588067    5515 generic.go:356] "Generic (PLEG): container finished" podID="860635a9-c37a-4f31-a4c3-b7d2a4ed5a20" containerID="30a48c03c0c50b9f32750db32518f27ed3ac60d11e833ae9ba91ebf475af761d" exitCode=0
Nov 03 03:04:58.588540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:58.588162    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerDied","Data":"30a48c03c0c50b9f32750db32518f27ed3ac60d11e833ae9ba91ebf475af761d"}
Nov 03 03:04:59.596412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.596378    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"6d9421c4653c5fad2ca1d4dff34bf2c2555f056a2d25da78f97d24a368c9283a"}
Nov 03 03:04:59.596840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.596465    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"4cdeef86992986d0c957102bf70fa80fcf221fd79ec84c6f3f607744f5e2196c"}
Nov 03 03:04:59.596840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.596494    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"544ce7a1f49eb0340576feb2fc6d17210788e2a380b9916c51d7f31ed832dd54"}
Nov 03 03:04:59.596840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.596518    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"3624ea3fe9c780e47a8a2a45915190cc0aa22b2f0ae2bc565db10e5cca24b270"}
Nov 03 03:04:59.596840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.596540    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"4a3d0c777e900c17abaf7d75eef7008ed5bd0ce59af1c33e3cdb83e5bd3b7f31"}
Nov 03 03:04:59.596840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.596616    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"860635a9-c37a-4f31-a4c3-b7d2a4ed5a20","Type":"ContainerStarted","Data":"5c508a63d3a2fcdc41bddf4947681ad0ac77d5d4b8a4cfd6a5b84d66bbc75b0f"}
Nov 03 03:04:59.729505 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:04:59.729455    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.713685918 podStartE2EDuration="3.729442518s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.7566ms" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:59.728531968 +0000 UTC m=+190.959424385" watchObservedRunningTime="2026-11-03 03:04:59.729442518 +0000 UTC m=+190.960334930"
Nov 03 03:05:04.170979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:04.170938    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:04.320461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:04.320428    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:07.683947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:07.683909    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:14.555339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.555304    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f4dbdb8d-tbsl5"]
Nov 03 03:05:14.558726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.558708    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.660000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.659969    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f4dbdb8d-tbsl5"]
Nov 03 03:05:14.691822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.691786    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-serving-cert\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.691987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.691832    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-oauth-config\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.691987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.691856    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-config\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.691987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.691946    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-service-ca\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.691987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.691972    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-oauth-serving-cert\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.692126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.691989    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-trusted-ca-bundle\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.692126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.692014    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b31bde2-6f27-49f8-b49c-a6fc196923f1-tmp\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.692126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.692056    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzd7w\" (UniqueName: \"kubernetes.io/projected/9b31bde2-6f27-49f8-b49c-a6fc196923f1-kube-api-access-xzd7w\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793297    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-service-ca\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793342    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-oauth-serving-cert\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793359    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-trusted-ca-bundle\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793374    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b31bde2-6f27-49f8-b49c-a6fc196923f1-tmp\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793392    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzd7w\" (UniqueName: \"kubernetes.io/projected/9b31bde2-6f27-49f8-b49c-a6fc196923f1-kube-api-access-xzd7w\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793437    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-serving-cert\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793485    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-oauth-config\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793517    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-config\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.793901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.793846    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b31bde2-6f27-49f8-b49c-a6fc196923f1-tmp\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.794287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.794263    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-oauth-serving-cert\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.794287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.794283    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-service-ca\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.794633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.794309    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-trusted-ca-bundle\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.794633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.794518    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-config\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.796152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.796131    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-serving-cert\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.796271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.796254    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b31bde2-6f27-49f8-b49c-a6fc196923f1-console-oauth-config\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.856727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.856646    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzd7w\" (UniqueName: \"kubernetes.io/projected/9b31bde2-6f27-49f8-b49c-a6fc196923f1-kube-api-access-xzd7w\") pod \"console-69f4dbdb8d-tbsl5\" (UID: \"9b31bde2-6f27-49f8-b49c-a6fc196923f1\") " pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:14.857002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.856972    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:14.869900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:14.869873    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:15.231972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:15.231922    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:15.242397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:15.242270    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f4dbdb8d-tbsl5"]
Nov 03 03:05:15.244956 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:05:15.244931    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b31bde2_6f27_49f8_b49c_a6fc196923f1.slice/crio-4288b6da4a1e4e22086f2ee2fed47aec764a417aa729574935877e0733a1c526 WatchSource:0}: Error finding container 4288b6da4a1e4e22086f2ee2fed47aec764a417aa729574935877e0733a1c526: Status 404 returned error can't find the container with id 4288b6da4a1e4e22086f2ee2fed47aec764a417aa729574935877e0733a1c526
Nov 03 03:05:15.377795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:15.377760    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:15.652219 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:15.652190    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f4dbdb8d-tbsl5" event={"ID":"9b31bde2-6f27-49f8-b49c-a6fc196923f1","Type":"ContainerStarted","Data":"638f0037bc40e7327ee97c51156c0a8a40d27f0e82496b81b3c6d1e6beb8f11b"}
Nov 03 03:05:15.652654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:15.652235    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f4dbdb8d-tbsl5" event={"ID":"9b31bde2-6f27-49f8-b49c-a6fc196923f1","Type":"ContainerStarted","Data":"4288b6da4a1e4e22086f2ee2fed47aec764a417aa729574935877e0733a1c526"}
Nov 03 03:05:15.824001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:15.823951    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69f4dbdb8d-tbsl5" podStartSLOduration=1.8239364 podStartE2EDuration="1.8239364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:05:15.773058122 +0000 UTC m=+207.003950539" watchObservedRunningTime="2026-11-03 03:05:15.8239364 +0000 UTC m=+207.054828815"
Nov 03 03:05:16.920972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:16.920935    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:17.068520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:17.068490    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:20.983459 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:05:20.983419    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:05:24.870569 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:24.870508    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:24.871057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:24.870590    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:24.875483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:24.875451    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:25.689135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:25.689108    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f4dbdb8d-tbsl5"
Nov 03 03:05:25.861901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:25.861869    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f946c5cf-rf2lw"]
Nov 03 03:05:28.999693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:28.999662    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:29.141682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:29.141651    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:48.689630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:48.689598    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:05:50.882021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:50.881953    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-59f946c5cf-rf2lw" podUID="47847104-e526-42fa-b564-61b6950d0562" containerName="console" containerID="cri-o://4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969" gracePeriod=15
Nov 03 03:05:50.984329 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:05:50.984283    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:05:51.116313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.116283    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:05:51.165706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165621    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165664    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp\" (UniqueName: \"kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165694    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165719    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165737    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165766    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165801    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.165944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.165839    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf\" (UniqueName: \"kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf\") pod \"47847104-e526-42fa-b564-61b6950d0562\" (UID: \"47847104-e526-42fa-b564-61b6950d0562\") "
Nov 03 03:05:51.166190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.166002    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:05:51.166276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.166251    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:05:51.166336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.166298    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:05:51.166393 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.166335    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:05:51.166393 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.166340    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:05:51.168113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.168089    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:05:51.168210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.168143    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:05:51.168210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.168193    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf" pod "47847104-e526-42fa-b564-61b6950d0562" (UID: "47847104-e526-42fa-b564-61b6950d0562"). InnerVolumeSpecName "kube-api-access-9vbhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:05:51.266904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266865    5515 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.266904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266901    5515 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47847104-e526-42fa-b564-61b6950d0562-tmp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.266904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266912    5515 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.267145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266921    5515 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47847104-e526-42fa-b564-61b6950d0562-console-oauth-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.267145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266930    5515 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-oauth-serving-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.267145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266939    5515 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-trusted-ca-bundle\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.267145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266946    5515 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47847104-e526-42fa-b564-61b6950d0562-console-config\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.267145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.266960    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbhf\" (UniqueName: \"kubernetes.io/projected/47847104-e526-42fa-b564-61b6950d0562-kube-api-access-9vbhf\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:05:51.774414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.774381    5515 generic.go:356] "Generic (PLEG): container finished" podID="47847104-e526-42fa-b564-61b6950d0562" containerID="4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969" exitCode=0
Nov 03 03:05:51.774414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.774418    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f946c5cf-rf2lw" event={"ID":"47847104-e526-42fa-b564-61b6950d0562","Type":"ContainerDied","Data":"4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969"}
Nov 03 03:05:51.774658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.774428    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f946c5cf-rf2lw"
Nov 03 03:05:51.774658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.774458    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f946c5cf-rf2lw" event={"ID":"47847104-e526-42fa-b564-61b6950d0562","Type":"ContainerDied","Data":"fe0409c8f83b13ac33a79619862fd37950743d07da716d7b759f7505a7705aa3"}
Nov 03 03:05:51.774658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.774478    5515 scope.go:122] "RemoveContainer" containerID="4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969"
Nov 03 03:05:51.783016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.782999    5515 scope.go:122] "RemoveContainer" containerID="4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969"
Nov 03 03:05:51.783307 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:05:51.783289    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969\": container with ID starting with 4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969 not found: ID does not exist" containerID="4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969"
Nov 03 03:05:51.783348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.783317    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969"} err="failed to get container status \"4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969\": rpc error: code = NotFound desc = could not find container \"4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969\": container with ID starting with 4b1f23fddc8e79f2f9038edda8ae2e0bb41cc7f37d8bb34df3406ca3af6ae969 not found: ID does not exist"
Nov 03 03:05:51.941802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.941770    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f946c5cf-rf2lw"]
Nov 03 03:05:51.992755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:51.992720    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59f946c5cf-rf2lw"]
Nov 03 03:05:52.890681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:05:52.890648    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47847104-e526-42fa-b564-61b6950d0562" path="/var/lib/kubelet/pods/47847104-e526-42fa-b564-61b6950d0562/volumes"
Nov 03 03:06:20.983120 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:06:20.983075    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:06:48.919493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:06:48.919472    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:06:48.920503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:06:48.920486    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:06:48.921856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:06:48.921832    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:06:48.922504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:06:48.922489    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:06:49.040428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:06:49.040396    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:06:50.983583 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:06:50.983530    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:07:20.983119 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:07:20.983082    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:07:47.118305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:47.118271    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:47.353822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:47.353782    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:47.608027 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:47.607998    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:49.262863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.262833    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"]
Nov 03 03:07:49.263237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.263090    5515 cpu_manager.go:414] "containerMap: removing container" podUID="47847104-e526-42fa-b564-61b6950d0562" containerName="console"
Nov 03 03:07:49.263237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.263100    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47847104-e526-42fa-b564-61b6950d0562" containerName="console"
Nov 03 03:07:49.263237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.263162    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="47847104-e526-42fa-b564-61b6950d0562" containerName="console"
Nov 03 03:07:49.265837 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.265820    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.283248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.283211    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:49.321440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.321416    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 03:07:49.323035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.323015    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 03:07:49.327139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.327125    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 03:07:49.339667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339643    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.339719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339673    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.339719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339696    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldzn\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.339787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339769    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.339889 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339873    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.339923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339900    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.339955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.339930    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.372401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.372377    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 03:07:49.381090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.381067    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"]
Nov 03 03:07:49.419398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.419369    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 03:07:49.423259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.423240    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 03:07:49.423335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.423307    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4d77v\""
Nov 03 03:07:49.439626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.439587    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:49.440674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440658    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.440732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440707    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.440732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440724    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.440831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440749    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldzn\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.440831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440767    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.440831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440788    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.440831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.440806    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.441719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.441578    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.443243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.443223    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.443339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.443306    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.443339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.443321    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.443422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.443306    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.503566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.503523    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldzn\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.503768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.503750    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hqwx2\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.576816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.576751    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:49.999179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:49.999147    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"]
Nov 03 03:07:50.002010 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:07:50.001983    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f73e8b_ba5f_4960_9891_ca104145f66f.slice/crio-371bf09c2fef44df35139e3c27914ce1b2624866a0a367a031e7764d54d08855 WatchSource:0}: Error finding container 371bf09c2fef44df35139e3c27914ce1b2624866a0a367a031e7764d54d08855: Status 404 returned error can't find the container with id 371bf09c2fef44df35139e3c27914ce1b2624866a0a367a031e7764d54d08855
Nov 03 03:07:50.165223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:50.165189    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" event={"ID":"f3f73e8b-ba5f-4960-9891-ca104145f66f","Type":"ContainerStarted","Data":"371bf09c2fef44df35139e3c27914ce1b2624866a0a367a031e7764d54d08855"}
Nov 03 03:07:50.717397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:50.717357    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:50.983002 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:07:50.982909    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:07:56.273381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:56.273350    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:07:56.273663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:56.273415    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:07:57.190452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.190416    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" event={"ID":"f3f73e8b-ba5f-4960-9891-ca104145f66f","Type":"ContainerStarted","Data":"4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885"}
Nov 03 03:07:57.190697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.190666    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:57.192061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.192038    5515 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-hqwx2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:07:57.192148 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.192092    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" podUID="f3f73e8b-ba5f-4960-9891-ca104145f66f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:07:57.421318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.421266    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" podStartSLOduration=2.152417458 podStartE2EDuration="8.421252075s" totalImagesPullingTime="6.268834617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:07:57.370367692 +0000 UTC m=+368.601260110" watchObservedRunningTime="2026-11-03 03:07:57.421252075 +0000 UTC m=+368.652144492"
Nov 03 03:07:57.587513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.587434    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:57.849527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:57.849452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:58.195258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:58.195229    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:07:58.302016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:58.301978    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:58.565288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:58.565260    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:07:58.800976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:07:58.800948    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:02.503836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:02.503787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:20.983328 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:08:20.983284    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:08:28.462680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:28.462636    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:29.251494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:29.251458    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:32.359377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:32.359324    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:33.325998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:33.325964    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:35.434411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:35.434379    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:35.631259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:35.631221    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:36.464180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:36.464145    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:36.843290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:36.843204    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:37.321873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:37.321837    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:37.627110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:37.627085    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:38.011319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:38.011238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:38.310811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:38.310726    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:39.819825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:39.819782    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:40.084651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.084567    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"]
Nov 03 03:08:40.093234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.093210    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.130961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.130938    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/429f5e9b-c85b-4b99-9a30-046ea71bde91-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.131113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.130980    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.131113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.130998    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/429f5e9b-c85b-4b99-9a30-046ea71bde91-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.131113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.131075    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.131238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.131115    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/429f5e9b-c85b-4b99-9a30-046ea71bde91-kube-api-access-qxlnw\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.131238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.131191    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.207826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.207796    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"]
Nov 03 03:08:40.220707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.220683    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 03:08:40.220707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.220709    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 03:08:40.220938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.220925    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 03:08:40.220975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.220940    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-slhgs\""
Nov 03 03:08:40.227376 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.227355    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 03:08:40.227469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.227386    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 03:08:40.231720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.231698    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/429f5e9b-c85b-4b99-9a30-046ea71bde91-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.231840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.231740    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.231840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.231757    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/429f5e9b-c85b-4b99-9a30-046ea71bde91-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.231840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.231781    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.231840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.231809    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/429f5e9b-c85b-4b99-9a30-046ea71bde91-kube-api-access-qxlnw\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.231840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.231840    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.232234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.232215    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/429f5e9b-c85b-4b99-9a30-046ea71bde91-manifests\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.234080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.234057    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/429f5e9b-c85b-4b99-9a30-046ea71bde91-tmp\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.234507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.234479    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.234601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.234504    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-metrics-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.234601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.234530    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/429f5e9b-c85b-4b99-9a30-046ea71bde91-webhook-cert\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.288465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.288436    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:40.296755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.296737    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/429f5e9b-c85b-4b99-9a30-046ea71bde91-kube-api-access-qxlnw\") pod \"opendatahub-operator-controller-manager-69c4cfd777-8nbr4\" (UID: \"429f5e9b-c85b-4b99-9a30-046ea71bde91\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.408285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.408247    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:40.584845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.584807    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:40.932662 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.932629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:40.952015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.951991    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"]
Nov 03 03:08:40.954359 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:08:40.954333    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429f5e9b_c85b_4b99_9a30_046ea71bde91.slice/crio-ebec1f721a04b1b8beb2738637654e01f508683f5fc76956a30f10ea27b4c8b7 WatchSource:0}: Error finding container ebec1f721a04b1b8beb2738637654e01f508683f5fc76956a30f10ea27b4c8b7: Status 404 returned error can't find the container with id ebec1f721a04b1b8beb2738637654e01f508683f5fc76956a30f10ea27b4c8b7
Nov 03 03:08:40.956095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:40.956073    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:08:41.335053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:41.334970    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4" event={"ID":"429f5e9b-c85b-4b99-9a30-046ea71bde91","Type":"ContainerStarted","Data":"ebec1f721a04b1b8beb2738637654e01f508683f5fc76956a30f10ea27b4c8b7"}
Nov 03 03:08:43.141280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:43.141246    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:43.696345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:43.696312    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:44.976709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:44.976671    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:45.149824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:45.149796    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:46.165058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.164982    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:46.355973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.355939    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4" event={"ID":"429f5e9b-c85b-4b99-9a30-046ea71bde91","Type":"ContainerStarted","Data":"2d6175f6cd5ee0e83fc4a0ce1a962ed6255d172cd4635e1675c8e5323f173c33"}
Nov 03 03:08:46.359496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.359474    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:46.779613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.779577    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"]
Nov 03 03:08:46.782939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.782919    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:08:46.871102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.871074    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"]
Nov 03 03:08:46.876727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.876706    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 03:08:46.876864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.876771    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 03:08:46.876975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.876956    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 03:08:46.877048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.877033    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mnnrb\""
Nov 03 03:08:46.884406 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.884384    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdkcd\" (UniqueName: \"kubernetes.io/projected/5c3cc519-a61c-4b59-905e-6c06aa56da13-kube-api-access-wdkcd\") pod \"dns-operator-controller-manager-8557cbdb84-m5qlx\" (UID: \"5c3cc519-a61c-4b59-905e-6c06aa56da13\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:08:46.984867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:46.984841    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdkcd\" (UniqueName: \"kubernetes.io/projected/5c3cc519-a61c-4b59-905e-6c06aa56da13-kube-api-access-wdkcd\") pod \"dns-operator-controller-manager-8557cbdb84-m5qlx\" (UID: \"5c3cc519-a61c-4b59-905e-6c06aa56da13\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:08:47.063504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:47.063434    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdkcd\" (UniqueName: \"kubernetes.io/projected/5c3cc519-a61c-4b59-905e-6c06aa56da13-kube-api-access-wdkcd\") pod \"dns-operator-controller-manager-8557cbdb84-m5qlx\" (UID: \"5c3cc519-a61c-4b59-905e-6c06aa56da13\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:08:47.094572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:47.094522    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:08:47.361011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:47.360980    5515 generic.go:356] "Generic (PLEG): container finished" podID="429f5e9b-c85b-4b99-9a30-046ea71bde91" containerID="2d6175f6cd5ee0e83fc4a0ce1a962ed6255d172cd4635e1675c8e5323f173c33" exitCode=0
Nov 03 03:08:47.361407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:47.361054    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4" event={"ID":"429f5e9b-c85b-4b99-9a30-046ea71bde91","Type":"ContainerDied","Data":"2d6175f6cd5ee0e83fc4a0ce1a962ed6255d172cd4635e1675c8e5323f173c33"}
Nov 03 03:08:47.472495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:47.472474    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"]
Nov 03 03:08:47.474958 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:08:47.474921    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3cc519_a61c_4b59_905e_6c06aa56da13.slice/crio-600820f0177271653846e5d304f5c8de2a8ab448e8d731be78052aba22258c50 WatchSource:0}: Error finding container 600820f0177271653846e5d304f5c8de2a8ab448e8d731be78052aba22258c50: Status 404 returned error can't find the container with id 600820f0177271653846e5d304f5c8de2a8ab448e8d731be78052aba22258c50
Nov 03 03:08:48.366603 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:48.366535    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx" event={"ID":"5c3cc519-a61c-4b59-905e-6c06aa56da13","Type":"ContainerStarted","Data":"600820f0177271653846e5d304f5c8de2a8ab448e8d731be78052aba22258c50"}
Nov 03 03:08:48.368468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:48.368443    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4" event={"ID":"429f5e9b-c85b-4b99-9a30-046ea71bde91","Type":"ContainerStarted","Data":"9178fa591981c0cbf56ca340d39297bb43526267705d9065c98a84f1d23ac4f0"}
Nov 03 03:08:48.368604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:48.368579    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:08:48.583717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:48.583610    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4" podStartSLOduration=3.404739827 podStartE2EDuration="9.583595547s" totalImagesPullingTime="5.764811443s" totalInitContainerRuntime="414.044277ms" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:48.581969133 +0000 UTC m=+419.812861552" watchObservedRunningTime="2026-11-03 03:08:48.583595547 +0000 UTC m=+419.814487964"
Nov 03 03:08:49.140236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.140205    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:49.327119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.327082    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:49.586216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.586146    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"]
Nov 03 03:08:49.589743 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.589718    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.661348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.661310    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-jln7g\""
Nov 03 03:08:49.665979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.665954    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"]
Nov 03 03:08:49.666246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.666198    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 03:08:49.666377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.666307    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 03:08:49.709453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.709427    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b11db886-d613-4007-aced-972b7d2c54c2-tmp\") pod \"jobset-operator-674cd59dc5-fxz84\" (UID: \"b11db886-d613-4007-aced-972b7d2c54c2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.709615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.709486    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwc2\" (UniqueName: \"kubernetes.io/projected/b11db886-d613-4007-aced-972b7d2c54c2-kube-api-access-ptwc2\") pod \"jobset-operator-674cd59dc5-fxz84\" (UID: \"b11db886-d613-4007-aced-972b7d2c54c2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.810375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.810338    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwc2\" (UniqueName: \"kubernetes.io/projected/b11db886-d613-4007-aced-972b7d2c54c2-kube-api-access-ptwc2\") pod \"jobset-operator-674cd59dc5-fxz84\" (UID: \"b11db886-d613-4007-aced-972b7d2c54c2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.810566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.810395    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b11db886-d613-4007-aced-972b7d2c54c2-tmp\") pod \"jobset-operator-674cd59dc5-fxz84\" (UID: \"b11db886-d613-4007-aced-972b7d2c54c2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.810769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.810752    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b11db886-d613-4007-aced-972b7d2c54c2-tmp\") pod \"jobset-operator-674cd59dc5-fxz84\" (UID: \"b11db886-d613-4007-aced-972b7d2c54c2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.904757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.904731    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwc2\" (UniqueName: \"kubernetes.io/projected/b11db886-d613-4007-aced-972b7d2c54c2-kube-api-access-ptwc2\") pod \"jobset-operator-674cd59dc5-fxz84\" (UID: \"b11db886-d613-4007-aced-972b7d2c54c2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.910613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.910589    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"
Nov 03 03:08:49.954028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:49.953995    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:50.292877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:50.292801    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84"]
Nov 03 03:08:50.296329 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:08:50.296305    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb11db886_d613_4007_aced_972b7d2c54c2.slice/crio-e0208a760bd963346921f8f2b6e66421a249dc0765675064265d8d43769800c3 WatchSource:0}: Error finding container e0208a760bd963346921f8f2b6e66421a249dc0765675064265d8d43769800c3: Status 404 returned error can't find the container with id e0208a760bd963346921f8f2b6e66421a249dc0765675064265d8d43769800c3
Nov 03 03:08:50.307714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:50.307672    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:50.378230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:50.378195    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84" event={"ID":"b11db886-d613-4007-aced-972b7d2c54c2","Type":"ContainerStarted","Data":"e0208a760bd963346921f8f2b6e66421a249dc0765675064265d8d43769800c3"}
Nov 03 03:08:50.983354 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:08:50.983322    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:08:51.084321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:51.084277    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:51.463612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:51.463572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:52.696378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:52.696345    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 03:08:53.393343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:53.393313    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx" event={"ID":"5c3cc519-a61c-4b59-905e-6c06aa56da13","Type":"ContainerStarted","Data":"0186e314ec7ad2ddb696e1e6dc079070c6462aedd40349e2efb855b4ff18762b"}
Nov 03 03:08:53.393567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:53.393476    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:08:53.518505 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:53.518449    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx" podStartSLOduration=2.352289767 podStartE2EDuration="7.518431297s" totalImagesPullingTime="5.16614153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:53.516301149 +0000 UTC m=+424.747193569" watchObservedRunningTime="2026-11-03 03:08:53.518431297 +0000 UTC m=+424.749323718"
Nov 03 03:08:55.267526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:55.267488    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:55.404175 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:55.404136    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84" event={"ID":"b11db886-d613-4007-aced-972b7d2c54c2","Type":"ContainerStarted","Data":"4a2adf6aba5ff3718d2f9eb570ecaa53ae1aeb5eeae5b3539c449a3743c6348b"}
Nov 03 03:08:55.521681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:55.521575    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:55.585011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:55.584961    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fxz84" podStartSLOduration=2.36255682 podStartE2EDuration="6.584947163s" totalImagesPullingTime="4.222390343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:55.584585914 +0000 UTC m=+426.815478335" watchObservedRunningTime="2026-11-03 03:08:55.584947163 +0000 UTC m=+426.815839581"
Nov 03 03:08:55.764424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:55.764369    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:08:59.375701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:08:59.375666    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69c4cfd777-8nbr4"
Nov 03 03:09:00.469542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:00.469481    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:00.647340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:00.647306    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:01.789570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:01.789521    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:02.075672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:02.075584    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:02.869310 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:02.869224    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:03.246393 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:03.246272    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:03.561982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:03.561855    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:04.401243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:04.401216    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-m5qlx"
Nov 03 03:09:05.576372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:05.576310    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:05.931155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:05.931125    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:05.989900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:05.989866    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"]
Nov 03 03:09:05.995586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:05.995544    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.059333 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.059309    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 03:09:06.103188 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.103156    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 03:09:06.103326 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.103200    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8prwk\""
Nov 03 03:09:06.104028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.104013    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"]
Nov 03 03:09:06.144717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.144687    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f8f7050-3fb9-4229-b35a-b5d831cdb957-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.144717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.144716    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8nk\" (UniqueName: \"kubernetes.io/projected/0f8f7050-3fb9-4229-b35a-b5d831cdb957-kube-api-access-6b8nk\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.144909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.144738    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f8f7050-3fb9-4229-b35a-b5d831cdb957-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.245595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.245487    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f8f7050-3fb9-4229-b35a-b5d831cdb957-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.245595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.245520    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8nk\" (UniqueName: \"kubernetes.io/projected/0f8f7050-3fb9-4229-b35a-b5d831cdb957-kube-api-access-6b8nk\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.245595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.245575    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f8f7050-3fb9-4229-b35a-b5d831cdb957-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.246357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.246294    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f8f7050-3fb9-4229-b35a-b5d831cdb957-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.248079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.248060    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f8f7050-3fb9-4229-b35a-b5d831cdb957-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.261108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.261080    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:06.308963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.308933    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8nk\" (UniqueName: \"kubernetes.io/projected/0f8f7050-3fb9-4229-b35a-b5d831cdb957-kube-api-access-6b8nk\") pod \"kuadrant-console-plugin-66c4fb575d-sj294\" (UID: \"0f8f7050-3fb9-4229-b35a-b5d831cdb957\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.457696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.457656    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:06.606591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.606490    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"
Nov 03 03:09:06.682735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:06.682706    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:07.074100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:07.074080    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294"]
Nov 03 03:09:07.076111 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:09:07.076076    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8f7050_3fb9_4229_b35a_b5d831cdb957.slice/crio-1e69e9cca5af3cc1c361bd2e54a71dc3e5c8107f4b08cbd02fa7286ef7fd018b WatchSource:0}: Error finding container 1e69e9cca5af3cc1c361bd2e54a71dc3e5c8107f4b08cbd02fa7286ef7fd018b: Status 404 returned error can't find the container with id 1e69e9cca5af3cc1c361bd2e54a71dc3e5c8107f4b08cbd02fa7286ef7fd018b
Nov 03 03:09:07.449986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:07.449947    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294" event={"ID":"0f8f7050-3fb9-4229-b35a-b5d831cdb957","Type":"ContainerStarted","Data":"1e69e9cca5af3cc1c361bd2e54a71dc3e5c8107f4b08cbd02fa7286ef7fd018b"}
Nov 03 03:09:07.538709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:07.538667    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:08.737967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:08.737938    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:11.274031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:11.273912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:11.626071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:11.626036    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:11.909625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:11.909537    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:12.509781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:12.509748    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:12.825953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:12.825856    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:13.177713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:13.177679    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:13.481266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:13.481186    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294" event={"ID":"0f8f7050-3fb9-4229-b35a-b5d831cdb957","Type":"ContainerStarted","Data":"078a6c1737293c23c520b1215f80b6eb12fa0166444f27f2e850fc02fa9aa428"}
Nov 03 03:09:13.640614 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:13.640539    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sj294" podStartSLOduration=2.8290585 podStartE2EDuration="8.640524907s" totalImagesPullingTime="5.811466407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:13.639517219 +0000 UTC m=+444.870409637" watchObservedRunningTime="2026-11-03 03:09:13.640524907 +0000 UTC m=+444.871417324"
Nov 03 03:09:14.206582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.206537    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"]
Nov 03 03:09:14.209141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.209121    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.217208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.217186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:14.277114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.277082    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"]
Nov 03 03:09:14.301444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.301413    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 03:09:14.301651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.301644    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 03:09:14.313015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.312997    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-tmp\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.313074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.313025    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-tls-certs\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.313146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.313127    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnn9\" (UniqueName: \"kubernetes.io/projected/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-kube-api-access-fpnn9\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.325647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.325617    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jgrrk\""
Nov 03 03:09:14.413828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.413788    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnn9\" (UniqueName: \"kubernetes.io/projected/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-kube-api-access-fpnn9\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.414001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.413854    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-tmp\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.414001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.413886    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-tls-certs\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.416287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.416265    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-tmp\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.416615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.416594    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-tls-certs\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.535653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.535581    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnn9\" (UniqueName: \"kubernetes.io/projected/4e3fdaae-993d-42e5-85fd-5bcdf96e67af-kube-api-access-fpnn9\") pod \"kube-auth-proxy-d5cd8cf48-sz5c2\" (UID: \"4e3fdaae-993d-42e5-85fd-5bcdf96e67af\") " pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.819988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.819910    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"
Nov 03 03:09:14.849688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:14.849653    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:15.212514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:15.212488    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2"]
Nov 03 03:09:15.215626 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:09:15.215591    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3fdaae_993d_42e5_85fd_5bcdf96e67af.slice/crio-eb725afd839dbbb95ee19a54cbe36a8d82cd8313379d5246391b0da1d10f7bb5 WatchSource:0}: Error finding container eb725afd839dbbb95ee19a54cbe36a8d82cd8313379d5246391b0da1d10f7bb5: Status 404 returned error can't find the container with id eb725afd839dbbb95ee19a54cbe36a8d82cd8313379d5246391b0da1d10f7bb5
Nov 03 03:09:15.220911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:15.220875    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:15.490116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:15.490035    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2" event={"ID":"4e3fdaae-993d-42e5-85fd-5bcdf96e67af","Type":"ContainerStarted","Data":"eb725afd839dbbb95ee19a54cbe36a8d82cd8313379d5246391b0da1d10f7bb5"}
Nov 03 03:09:15.561072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:15.561019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:16.034270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:16.034202    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:17.005539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:17.005420    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:17.324419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:17.324348    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:18.102275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:18.102245    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:18.433451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:18.433418    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:18.505058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:18.505025    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2" event={"ID":"4e3fdaae-993d-42e5-85fd-5bcdf96e67af","Type":"ContainerStarted","Data":"ada1c62a6b7a1af46c62f7fbb2c3aa345e86eb2935425c98a5df3c8240544132"}
Nov 03 03:09:18.636493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:18.636424    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d5cd8cf48-sz5c2" podStartSLOduration=1.7438011979999999 podStartE2EDuration="4.636404065s" totalImagesPullingTime="2.892602867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:18.631741584 +0000 UTC m=+449.862634002" watchObservedRunningTime="2026-11-03 03:09:18.636404065 +0000 UTC m=+449.867296486"
Nov 03 03:09:18.731133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:18.731058    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:19.474093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.474054    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:09:19.476355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.476332    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:19.541988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.541955    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 03:09:19.590630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.590591    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:09:19.667090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.667058    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrj5k\" (UniqueName: \"kubernetes.io/projected/83d03a51-825c-477b-9fcc-e5af9421d387-kube-api-access-nrj5k\") pod \"limitador-limitador-7c9b87d574-lhs8n\" (UID: \"83d03a51-825c-477b-9fcc-e5af9421d387\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:19.667276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.667112    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83d03a51-825c-477b-9fcc-e5af9421d387-config-file\") pod \"limitador-limitador-7c9b87d574-lhs8n\" (UID: \"83d03a51-825c-477b-9fcc-e5af9421d387\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:19.767828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.767738    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrj5k\" (UniqueName: \"kubernetes.io/projected/83d03a51-825c-477b-9fcc-e5af9421d387-kube-api-access-nrj5k\") pod \"limitador-limitador-7c9b87d574-lhs8n\" (UID: \"83d03a51-825c-477b-9fcc-e5af9421d387\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:19.767828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.767812    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83d03a51-825c-477b-9fcc-e5af9421d387-config-file\") pod \"limitador-limitador-7c9b87d574-lhs8n\" (UID: \"83d03a51-825c-477b-9fcc-e5af9421d387\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:19.768468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.768447    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83d03a51-825c-477b-9fcc-e5af9421d387-config-file\") pod \"limitador-limitador-7c9b87d574-lhs8n\" (UID: \"83d03a51-825c-477b-9fcc-e5af9421d387\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:19.788113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.788079    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:09:19.830475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:19.830449    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrj5k\" (UniqueName: \"kubernetes.io/projected/83d03a51-825c-477b-9fcc-e5af9421d387-kube-api-access-nrj5k\") pod \"limitador-limitador-7c9b87d574-lhs8n\" (UID: \"83d03a51-825c-477b-9fcc-e5af9421d387\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:20.087080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:20.086986    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:20.477540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:20.477503    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:09:20.481367 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:09:20.481344    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d03a51_825c_477b_9fcc_e5af9421d387.slice/crio-4c36fc5274a4ae10d7f80db3cd2c3c4f7e1a73940db5611ba9e458327c9bba5d WatchSource:0}: Error finding container 4c36fc5274a4ae10d7f80db3cd2c3c4f7e1a73940db5611ba9e458327c9bba5d: Status 404 returned error can't find the container with id 4c36fc5274a4ae10d7f80db3cd2c3c4f7e1a73940db5611ba9e458327c9bba5d
Nov 03 03:09:20.513462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:20.513426    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n" event={"ID":"83d03a51-825c-477b-9fcc-e5af9421d387","Type":"ContainerStarted","Data":"4c36fc5274a4ae10d7f80db3cd2c3c4f7e1a73940db5611ba9e458327c9bba5d"}
Nov 03 03:09:20.982218 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:09:20.982185    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:09:22.356782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:22.356749    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:22.522628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:22.522593    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n" event={"ID":"83d03a51-825c-477b-9fcc-e5af9421d387","Type":"ContainerStarted","Data":"e9599cb07594a5bffdcbc43cb08483f9c42b136cd0acea64f59f2937c5256cd1"}
Nov 03 03:09:22.522824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:22.522713    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:22.580381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:22.580352    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:22.643489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:22.643432    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n" podStartSLOduration=1.692310891 podStartE2EDuration="3.643418549s" totalImagesPullingTime="1.951107658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:22.642678886 +0000 UTC m=+453.873571304" watchObservedRunningTime="2026-11-03 03:09:22.643418549 +0000 UTC m=+453.874310960"
Nov 03 03:09:28.842942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:28.842912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:28.996887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:28.996853    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:33.528273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:33.528244    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"
Nov 03 03:09:40.057176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:40.057141    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:40.361976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:40.361949    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:40.858493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:40.858447    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:45.236220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:45.236183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:45.386914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:45.386879    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:50.121352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:50.121317    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:50.983383 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:09:50.983329    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:09:51.945350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:51.945314    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:09:53.593786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:09:53.593753    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:02.744828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:02.744792    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:02.898927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:02.898896    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:03.051153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:03.051060    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:03.222601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:03.222570    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:09.694328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:09.694291    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:09.853266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:09.853231    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:12.892050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:12.892008    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:13.081731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.081702    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:13.744798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.744764    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:13.824380 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.824348    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-discovery-845d48d7b4-6k262"]
Nov 03 03:10:13.826724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.826705    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:13.920785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.920752    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5xz\" (UniqueName: \"kubernetes.io/projected/7ccb2946-a5ce-4a22-b5eb-301c6373c834-kube-api-access-xd5xz\") pod \"maas-discovery-845d48d7b4-6k262\" (UID: \"7ccb2946-a5ce-4a22-b5eb-301c6373c834\") " pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:13.921161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.920804    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccb2946-a5ce-4a22-b5eb-301c6373c834-tls-certs\") pod \"maas-discovery-845d48d7b4-6k262\" (UID: \"7ccb2946-a5ce-4a22-b5eb-301c6373c834\") " pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:13.930697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.930666    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 03:10:13.930697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.930680    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 03:10:13.930843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.930681    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-discovery-dockercfg-pqb7f\""
Nov 03 03:10:13.930843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.930720    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-discovery-tls\""
Nov 03 03:10:13.931734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.931715    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-discovery-845d48d7b4-6k262"]
Nov 03 03:10:13.932756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:13.932724    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:14.021295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.021214    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5xz\" (UniqueName: \"kubernetes.io/projected/7ccb2946-a5ce-4a22-b5eb-301c6373c834-kube-api-access-xd5xz\") pod \"maas-discovery-845d48d7b4-6k262\" (UID: \"7ccb2946-a5ce-4a22-b5eb-301c6373c834\") " pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:14.021295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.021271    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccb2946-a5ce-4a22-b5eb-301c6373c834-tls-certs\") pod \"maas-discovery-845d48d7b4-6k262\" (UID: \"7ccb2946-a5ce-4a22-b5eb-301c6373c834\") " pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:14.023785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.023762    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ccb2946-a5ce-4a22-b5eb-301c6373c834-tls-certs\") pod \"maas-discovery-845d48d7b4-6k262\" (UID: \"7ccb2946-a5ce-4a22-b5eb-301c6373c834\") " pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:14.083891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.083861    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5xz\" (UniqueName: \"kubernetes.io/projected/7ccb2946-a5ce-4a22-b5eb-301c6373c834-kube-api-access-xd5xz\") pod \"maas-discovery-845d48d7b4-6k262\" (UID: \"7ccb2946-a5ce-4a22-b5eb-301c6373c834\") " pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:14.138112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.138088    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:14.506073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.506043    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-discovery-845d48d7b4-6k262"]
Nov 03 03:10:14.512141 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:10:14.512109    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccb2946_a5ce_4a22_b5eb_301c6373c834.slice/crio-bce59cdd0abaf3e979e5428e0f345226de10e10d50655947fcd2da8d5c8dd996 WatchSource:0}: Error finding container bce59cdd0abaf3e979e5428e0f345226de10e10d50655947fcd2da8d5c8dd996: Status 404 returned error can't find the container with id bce59cdd0abaf3e979e5428e0f345226de10e10d50655947fcd2da8d5c8dd996
Nov 03 03:10:14.731561 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:14.731511    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-discovery-845d48d7b4-6k262" event={"ID":"7ccb2946-a5ce-4a22-b5eb-301c6373c834","Type":"ContainerStarted","Data":"bce59cdd0abaf3e979e5428e0f345226de10e10d50655947fcd2da8d5c8dd996"}
Nov 03 03:10:20.757072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:20.757036    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-discovery-845d48d7b4-6k262" event={"ID":"7ccb2946-a5ce-4a22-b5eb-301c6373c834","Type":"ContainerStarted","Data":"379ded17451370b7bbf3f6fb50a774863d576c0d93dc0a9612047a5cb16a70c1"}
Nov 03 03:10:20.757446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:20.757158    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:20.876894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:20.876848    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-discovery-845d48d7b4-6k262" podStartSLOduration=1.8526973359999999 podStartE2EDuration="7.87683232s" totalImagesPullingTime="6.024134984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:20.87612477 +0000 UTC m=+512.107017169" watchObservedRunningTime="2026-11-03 03:10:20.87683232 +0000 UTC m=+512.107724773"
Nov 03 03:10:20.983092 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:10:20.983058    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:10:26.766759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:26.766724    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-discovery-845d48d7b4-6k262"
Nov 03 03:10:27.993653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:27.993617    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:28.139627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:28.139590    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:40.971099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:40.971066    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:41.128092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:41.128060    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:43.160178 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:43.160146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:43.308857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:43.308829    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:43.455953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:43.455867    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:45.519008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.518979    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"]
Nov 03 03:10:45.522486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.522468    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562508    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f392881-5978-44e2-8284-ff5d07d80563-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562586    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562660    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562713    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5xk\" (UniqueName: \"kubernetes.io/projected/0f392881-5978-44e2-8284-ff5d07d80563-kube-api-access-hw5xk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562741    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f392881-5978-44e2-8284-ff5d07d80563-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562779    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f392881-5978-44e2-8284-ff5d07d80563-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.562860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.562843    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.624495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.624459    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"]
Nov 03 03:10:45.663383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663348    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.663383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663384    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.663691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663406    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5xk\" (UniqueName: \"kubernetes.io/projected/0f392881-5978-44e2-8284-ff5d07d80563-kube-api-access-hw5xk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.663691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663436    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f392881-5978-44e2-8284-ff5d07d80563-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.663691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663477    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f392881-5978-44e2-8284-ff5d07d80563-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.663691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663508    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.663691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.663579    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f392881-5978-44e2-8284-ff5d07d80563-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.664518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.664489    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f392881-5978-44e2-8284-ff5d07d80563-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.666076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.666048    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f392881-5978-44e2-8284-ff5d07d80563-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.666187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.666084    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.666277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.666256    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.666327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.666256    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f392881-5978-44e2-8284-ff5d07d80563-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.726094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.726060    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f392881-5978-44e2-8284-ff5d07d80563-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.726240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.726069    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5xk\" (UniqueName: \"kubernetes.io/projected/0f392881-5978-44e2-8284-ff5d07d80563-kube-api-access-hw5xk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zb5pl\" (UID: \"0f392881-5978-44e2-8284-ff5d07d80563\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:45.833440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:45.833360    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:46.278084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.278060    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"]
Nov 03 03:10:46.280385 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:10:46.280357    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f392881_5978_44e2_8284_ff5d07d80563.slice/crio-9fe24775c48194fd2acb66db391ae579b3242ac21fad91ff29128742a9c33281 WatchSource:0}: Error finding container 9fe24775c48194fd2acb66db391ae579b3242ac21fad91ff29128742a9c33281: Status 404 returned error can't find the container with id 9fe24775c48194fd2acb66db391ae579b3242ac21fad91ff29128742a9c33281
Nov 03 03:10:46.282362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.282333    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:10:46.282430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.282390    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:10:46.861638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.861598    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl" event={"ID":"0f392881-5978-44e2-8284-ff5d07d80563","Type":"ContainerStarted","Data":"ee6695b279c074dbe42b5f7c9607e31a6dea8ea987f28e72d30aead446979ebd"}
Nov 03 03:10:46.862051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.861651    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl" event={"ID":"0f392881-5978-44e2-8284-ff5d07d80563","Type":"ContainerStarted","Data":"9fe24775c48194fd2acb66db391ae579b3242ac21fad91ff29128742a9c33281"}
Nov 03 03:10:46.862051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.861684    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:46.993091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:46.993042    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl" podStartSLOduration=1.9930271990000001 podStartE2EDuration="1.993027199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:46.99103607 +0000 UTC m=+538.221928487" watchObservedRunningTime="2026-11-03 03:10:46.993027199 +0000 UTC m=+538.223919617"
Nov 03 03:10:47.868160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:47.868135    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zb5pl"
Nov 03 03:10:48.094987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.094953    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"]
Nov 03 03:10:48.095168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.095146    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" podUID="f3f73e8b-ba5f-4960-9891-ca104145f66f" containerName="discovery" containerID="cri-o://4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885" gracePeriod=30
Nov 03 03:10:48.339085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.339060    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:10:48.389184 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389156    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389220    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389244    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389293    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389323    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389360    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389388    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig\") pod \"f3f73e8b-ba5f-4960-9891-ca104145f66f\" (UID: \"f3f73e8b-ba5f-4960-9891-ca104145f66f\") "
Nov 03 03:10:48.389783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.389733    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:10:48.392145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.392105    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:10:48.392145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.392139    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:10:48.392264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.392143    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:10:48.392264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.392184    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "kube-api-access-mldzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:10:48.392264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.392215    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:10:48.392264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.392229    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert" pod "f3f73e8b-ba5f-4960-9891-ca104145f66f" (UID: "f3f73e8b-ba5f-4960-9891-ca104145f66f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490397    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490422    5515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-dns-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490431    5515 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f73e8b-ba5f-4960-9891-ca104145f66f-local-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490440    5515 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-csr-ca-configmap\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490450    5515 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-cacerts\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490459    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldzn\" (UniqueName: \"kubernetes.io/projected/f3f73e8b-ba5f-4960-9891-ca104145f66f-kube-api-access-mldzn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.490464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.490467    5515 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3f73e8b-ba5f-4960-9891-ca104145f66f-istio-kubeconfig\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:10:48.872274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.872238    5515 generic.go:356] "Generic (PLEG): container finished" podID="f3f73e8b-ba5f-4960-9891-ca104145f66f" containerID="4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885" exitCode=0
Nov 03 03:10:48.872748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.872273    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" event={"ID":"f3f73e8b-ba5f-4960-9891-ca104145f66f","Type":"ContainerDied","Data":"4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885"}
Nov 03 03:10:48.872748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.872303    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"
Nov 03 03:10:48.872748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.872337    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2" event={"ID":"f3f73e8b-ba5f-4960-9891-ca104145f66f","Type":"ContainerDied","Data":"371bf09c2fef44df35139e3c27914ce1b2624866a0a367a031e7764d54d08855"}
Nov 03 03:10:48.872748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.872354    5515 scope.go:122] "RemoveContainer" containerID="4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885"
Nov 03 03:10:48.883601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.883582    5515 scope.go:122] "RemoveContainer" containerID="4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885"
Nov 03 03:10:48.883877 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:10:48.883860    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885\": container with ID starting with 4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885 not found: ID does not exist" containerID="4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885"
Nov 03 03:10:48.883925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:48.883886    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885"} err="failed to get container status \"4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885\": rpc error: code = NotFound desc = could not find container \"4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885\": container with ID starting with 4d28542c72fa588488f87b0ce7da634fb9756bd3dce681fd65468a0916bd7885 not found: ID does not exist"
Nov 03 03:10:49.046164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:49.046126    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"]
Nov 03 03:10:49.097039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:49.097011    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hqwx2"]
Nov 03 03:10:49.792772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:49.792737    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:49.969000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:49.968951    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:50.122479 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.122454    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:50.189956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.189923    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:10:50.392233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.392146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:50.540605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.540571    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:50.694740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.694659    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:50.747276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.747247    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7tdsv"]
Nov 03 03:10:50.747586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.747571    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f3f73e8b-ba5f-4960-9891-ca104145f66f" containerName="discovery"
Nov 03 03:10:50.747638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.747587    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f73e8b-ba5f-4960-9891-ca104145f66f" containerName="discovery"
Nov 03 03:10:50.747679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.747665    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f73e8b-ba5f-4960-9891-ca104145f66f" containerName="discovery"
Nov 03 03:10:50.751972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.751952    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.808260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.808233    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/108ea924-255e-45c4-85a3-deaa09ddb1f0-cert\") pod \"kserve-controller-manager-776cc66d47-7tdsv\" (UID: \"108ea924-255e-45c4-85a3-deaa09ddb1f0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.808441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.808273    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bzr\" (UniqueName: \"kubernetes.io/projected/108ea924-255e-45c4-85a3-deaa09ddb1f0-kube-api-access-b4bzr\") pod \"kserve-controller-manager-776cc66d47-7tdsv\" (UID: \"108ea924-255e-45c4-85a3-deaa09ddb1f0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.813995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.813969    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7tdsv"]
Nov 03 03:10:50.815434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.815413    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jpsd6\""
Nov 03 03:10:50.815592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.815450    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 03:10:50.859654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.859618    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:50.891789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.891757    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f73e8b-ba5f-4960-9891-ca104145f66f" path="/var/lib/kubelet/pods/f3f73e8b-ba5f-4960-9891-ca104145f66f/volumes"
Nov 03 03:10:50.909408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.909377    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/108ea924-255e-45c4-85a3-deaa09ddb1f0-cert\") pod \"kserve-controller-manager-776cc66d47-7tdsv\" (UID: \"108ea924-255e-45c4-85a3-deaa09ddb1f0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.909521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.909418    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bzr\" (UniqueName: \"kubernetes.io/projected/108ea924-255e-45c4-85a3-deaa09ddb1f0-kube-api-access-b4bzr\") pod \"kserve-controller-manager-776cc66d47-7tdsv\" (UID: \"108ea924-255e-45c4-85a3-deaa09ddb1f0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.912100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.912066    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/108ea924-255e-45c4-85a3-deaa09ddb1f0-cert\") pod \"kserve-controller-manager-776cc66d47-7tdsv\" (UID: \"108ea924-255e-45c4-85a3-deaa09ddb1f0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.974757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:50.974682    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bzr\" (UniqueName: \"kubernetes.io/projected/108ea924-255e-45c4-85a3-deaa09ddb1f0-kube-api-access-b4bzr\") pod \"kserve-controller-manager-776cc66d47-7tdsv\" (UID: \"108ea924-255e-45c4-85a3-deaa09ddb1f0\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:50.986384 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:10:50.986345    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:10:51.014039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:51.014007    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:51.064388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:51.064357    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:51.436058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:51.435911    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7tdsv"]
Nov 03 03:10:51.438700 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:10:51.438671    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108ea924_255e_45c4_85a3_deaa09ddb1f0.slice/crio-5c6732e01c315bcfa194d67eb2de1d3f5db17ae7167ea94c244ae18e9f0f3df7 WatchSource:0}: Error finding container 5c6732e01c315bcfa194d67eb2de1d3f5db17ae7167ea94c244ae18e9f0f3df7: Status 404 returned error can't find the container with id 5c6732e01c315bcfa194d67eb2de1d3f5db17ae7167ea94c244ae18e9f0f3df7
Nov 03 03:10:51.886018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:51.885987    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv" event={"ID":"108ea924-255e-45c4-85a3-deaa09ddb1f0","Type":"ContainerStarted","Data":"5c6732e01c315bcfa194d67eb2de1d3f5db17ae7167ea94c244ae18e9f0f3df7"}
Nov 03 03:10:52.164364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:52.164256    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:54.900857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:54.900823    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv" event={"ID":"108ea924-255e-45c4-85a3-deaa09ddb1f0","Type":"ContainerStarted","Data":"2faf816910471c6b8a99288a35c1ed6d7829f9448008f757868d0501d07959f9"}
Nov 03 03:10:54.901284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:54.900937    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:10:55.020709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:55.020618    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv" podStartSLOduration=1.795641604 podStartE2EDuration="5.02060266s" totalImagesPullingTime="3.224961056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:10:55.019868927 +0000 UTC m=+546.250761345" watchObservedRunningTime="2026-11-03 03:10:55.02060266 +0000 UTC m=+546.251495078"
Nov 03 03:10:55.567208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:55.567170    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:55.842853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:55.842775    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:56.168305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.168273    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"]
Nov 03 03:10:56.171486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.171470    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.259153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.259122    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.259305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.259169    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8s7\" (UniqueName: \"kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.259305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.259192    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.259382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.259306    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.275720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.275686    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:10:56.275720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.275711    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:10:56.275913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.275850    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 03:10:56.275954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.275935    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 03:10:56.276013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.275998    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcxk\""
Nov 03 03:10:56.277731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.277714    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"]
Nov 03 03:10:56.359717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.359684    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.359892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.359733    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.359892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.359766    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8s7\" (UniqueName: \"kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.359892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.359871    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.360495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.360444    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.362190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.362172    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.362377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.362356    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.421684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.421620    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8s7\" (UniqueName: \"kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7\") pod \"maas-api-55cd9fdc56-bg452\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") " pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.483319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.483287    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:10:56.861309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.861201    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"]
Nov 03 03:10:56.863895 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:10:56.863865    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7332092d_faf6_464b_a496_e4a906ee83af.slice/crio-945bca5b26ef1f73ccdbcbeeb286df62324aaa92e580aacee81636a5d3101e3d WatchSource:0}: Error finding container 945bca5b26ef1f73ccdbcbeeb286df62324aaa92e580aacee81636a5d3101e3d: Status 404 returned error can't find the container with id 945bca5b26ef1f73ccdbcbeeb286df62324aaa92e580aacee81636a5d3101e3d
Nov 03 03:10:56.911329 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:56.911294    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452" event={"ID":"7332092d-faf6-464b-a496-e4a906ee83af","Type":"ContainerStarted","Data":"945bca5b26ef1f73ccdbcbeeb286df62324aaa92e580aacee81636a5d3101e3d"}
Nov 03 03:10:57.557761 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:57.557727    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:57.700218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:57.700178    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:57.850334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:57.850259    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:57.992727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:57.992697    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:58.136457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:58.136426    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:58.287162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:58.287123    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:58.434804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:58.434730    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:58.584533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:58.584495    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:58.735136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:58.735052    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:10:59.925508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:59.925476    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452" event={"ID":"7332092d-faf6-464b-a496-e4a906ee83af","Type":"ContainerStarted","Data":"436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e"}
Nov 03 03:10:59.925978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:10:59.925606    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:11:00.048953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:00.048904    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452" podStartSLOduration=1.771443599 podStartE2EDuration="4.048891099s" totalImagesPullingTime="2.2774475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:11:00.048219018 +0000 UTC m=+551.279111435" watchObservedRunningTime="2026-11-03 03:11:00.048891099 +0000 UTC m=+551.279783517"
Nov 03 03:11:05.936072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:05.936040    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 03:11:20.983106 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:11:20.983071    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:11:25.911814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:25.911774    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7tdsv"
Nov 03 03:11:48.950292 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:48.950266    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:11:48.950811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:48.950770    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:11:48.952291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:48.952270    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:11:48.952909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:48.952889    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:11:50.833818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:50.833784    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:11:50.983449 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:11:50.983416    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:11:59.378220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:59.378186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:11:59.531376 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:11:59.531345    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:02.968173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:02.968099    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:20.983705 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:12:20.983668    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:12:35.748376 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:35.748281    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:35.921444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:35.921405    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:43.403371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.403333    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:43.596029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.595998    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:43.827006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.826916    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"]
Nov 03 03:12:43.829453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.829434    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:43.844637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.844614    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg946\" (UniqueName: \"kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946\") pod \"payload-pre-processing-7b98bb678c-vwq57\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:43.844738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.844663    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vwq57\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:43.933611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.933570    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 03:12:43.933611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.933575    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2h5gl\""
Nov 03 03:12:43.934775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.934757    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"]
Nov 03 03:12:43.945112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.945092    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg946\" (UniqueName: \"kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946\") pod \"payload-pre-processing-7b98bb678c-vwq57\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:43.945326 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.945298    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vwq57\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:43.945975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:43.945953    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-vwq57\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:44.008377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:44.008343    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg946\" (UniqueName: \"kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946\") pod \"payload-pre-processing-7b98bb678c-vwq57\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:44.139817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:44.139784    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:44.509474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:44.509452    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"]
Nov 03 03:12:44.512419 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:12:44.512392    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5844b997_7f93_44e1_b813_ada656a64412.slice/crio-5ef4241b2270c55774a7ea5575c987cb42cb9382eb86fb3520b4bc3e8c4c1896 WatchSource:0}: Error finding container 5ef4241b2270c55774a7ea5575c987cb42cb9382eb86fb3520b4bc3e8c4c1896: Status 404 returned error can't find the container with id 5ef4241b2270c55774a7ea5575c987cb42cb9382eb86fb3520b4bc3e8c4c1896
Nov 03 03:12:45.358197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:45.358166    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57" event={"ID":"5844b997-7f93-44e1-b813-ada656a64412","Type":"ContainerStarted","Data":"5ef4241b2270c55774a7ea5575c987cb42cb9382eb86fb3520b4bc3e8c4c1896"}
Nov 03 03:12:47.368951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:47.368916    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57" event={"ID":"5844b997-7f93-44e1-b813-ada656a64412","Type":"ContainerStarted","Data":"34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846"}
Nov 03 03:12:47.369385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:47.369022    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:12:47.489956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:47.489902    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57" podStartSLOduration=1.7428396849999999 podStartE2EDuration="4.489887116s" totalImagesPullingTime="2.747047431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:47.489114572 +0000 UTC m=+658.720006989" watchObservedRunningTime="2026-11-03 03:12:47.489887116 +0000 UTC m=+658.720779534"
Nov 03 03:12:50.983323 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:12:50.983288    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:12:51.230092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.230048    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:12:51.447866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.447829    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"]
Nov 03 03:12:51.450218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.450200    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.503888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.503857    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 03:12:51.505182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.505161    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ba1b83-66ba-4138-ae86-98b9dd588eed-metrics-certs\") pod \"ai-gateway-controller-6d99f9cf84-h9p42\" (UID: \"93ba1b83-66ba-4138-ae86-98b9dd588eed\") " pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.505304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.505284    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxcz8\" (UniqueName: \"kubernetes.io/projected/93ba1b83-66ba-4138-ae86-98b9dd588eed-kube-api-access-dxcz8\") pod \"ai-gateway-controller-6d99f9cf84-h9p42\" (UID: \"93ba1b83-66ba-4138-ae86-98b9dd588eed\") " pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.554679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.554647    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-skt72\""
Nov 03 03:12:51.554679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.554681    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 03:12:51.555784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.555768    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"]
Nov 03 03:12:51.605893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.605858    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ba1b83-66ba-4138-ae86-98b9dd588eed-metrics-certs\") pod \"ai-gateway-controller-6d99f9cf84-h9p42\" (UID: \"93ba1b83-66ba-4138-ae86-98b9dd588eed\") " pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.606157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.605927    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxcz8\" (UniqueName: \"kubernetes.io/projected/93ba1b83-66ba-4138-ae86-98b9dd588eed-kube-api-access-dxcz8\") pod \"ai-gateway-controller-6d99f9cf84-h9p42\" (UID: \"93ba1b83-66ba-4138-ae86-98b9dd588eed\") " pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.608635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.608611    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ba1b83-66ba-4138-ae86-98b9dd588eed-metrics-certs\") pod \"ai-gateway-controller-6d99f9cf84-h9p42\" (UID: \"93ba1b83-66ba-4138-ae86-98b9dd588eed\") " pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.668515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.668488    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxcz8\" (UniqueName: \"kubernetes.io/projected/93ba1b83-66ba-4138-ae86-98b9dd588eed-kube-api-access-dxcz8\") pod \"ai-gateway-controller-6d99f9cf84-h9p42\" (UID: \"93ba1b83-66ba-4138-ae86-98b9dd588eed\") " pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:51.761400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:51.761309    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:52.132782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:52.132755    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"]
Nov 03 03:12:52.135205 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:12:52.135182    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ba1b83_66ba_4138_ae86_98b9dd588eed.slice/crio-c2363185e78af28324c5681ff24b3378a4dab4519c8307019212bc1f55580834 WatchSource:0}: Error finding container c2363185e78af28324c5681ff24b3378a4dab4519c8307019212bc1f55580834: Status 404 returned error can't find the container with id c2363185e78af28324c5681ff24b3378a4dab4519c8307019212bc1f55580834
Nov 03 03:12:52.390373 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:52.390293    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42" event={"ID":"93ba1b83-66ba-4138-ae86-98b9dd588eed","Type":"ContainerStarted","Data":"c2363185e78af28324c5681ff24b3378a4dab4519c8307019212bc1f55580834"}
Nov 03 03:12:55.408823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:55.408785    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42" event={"ID":"93ba1b83-66ba-4138-ae86-98b9dd588eed","Type":"ContainerStarted","Data":"e610b89d980e094d357230cf44bde22386a7a8269e28efbc4d2b09900fe9f65b"}
Nov 03 03:12:55.409311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:55.408923    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:12:55.527992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:55.527941    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42" podStartSLOduration=2.096716502 podStartE2EDuration="4.527928355s" totalImagesPullingTime="2.431211853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:12:55.527728846 +0000 UTC m=+666.758621265" watchObservedRunningTime="2026-11-03 03:12:55.527928355 +0000 UTC m=+666.758820772"
Nov 03 03:12:58.375305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:12:58.375281    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 03:13:06.415126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:06.415096    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6d99f9cf84-h9p42"
Nov 03 03:13:08.391953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:08.391926    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:08.944466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:08.944434    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:10.367805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:10.367771    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:10.519454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:10.519424    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:18.206167 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:18.206138    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:20.983508 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:13:20.983472    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:13:21.797469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:21.797435    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:13:25.821685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.821642    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"]
Nov 03 03:13:25.824154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.824134    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:25.877729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.877700    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 03:13:25.912381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.912353    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:25.928343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.928319    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b5ckh\""
Nov 03 03:13:25.928492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.928350    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 03:13:25.928492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.928362    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 03:13:25.929698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:25.929678    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"]
Nov 03 03:13:26.009666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.009629    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.009666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.009669    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.009886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.009705    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.009886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.009751    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c25k\" (UniqueName: \"kubernetes.io/projected/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-kube-api-access-8c25k\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.009886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.009842    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.009886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.009869    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110299    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110352    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110375    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110398    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110420    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c25k\" (UniqueName: \"kubernetes.io/projected/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-kube-api-access-8c25k\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110495    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110783    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110802    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.110852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.110828    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.112928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.112901    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.113053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.113036    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.172809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.172781    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c25k\" (UniqueName: \"kubernetes.io/projected/d373288c-3e8d-4c1d-b7a7-90eb5dd2e488-kube-api-access-8c25k\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-p8clw\" (UID: \"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.435497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.435411    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:26.806330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:26.806258    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"]
Nov 03 03:13:26.809481 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:13:26.809451    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd373288c_3e8d_4c1d_b7a7_90eb5dd2e488.slice/crio-c803dc3a1fab1616e53f49c71d3484fcd920c8c07cb9d3517703f475fb2bb912 WatchSource:0}: Error finding container c803dc3a1fab1616e53f49c71d3484fcd920c8c07cb9d3517703f475fb2bb912: Status 404 returned error can't find the container with id c803dc3a1fab1616e53f49c71d3484fcd920c8c07cb9d3517703f475fb2bb912
Nov 03 03:13:27.540681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:27.540642    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw" event={"ID":"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488","Type":"ContainerStarted","Data":"c803dc3a1fab1616e53f49c71d3484fcd920c8c07cb9d3517703f475fb2bb912"}
Nov 03 03:13:28.546732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:28.546640    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw" event={"ID":"d373288c-3e8d-4c1d-b7a7-90eb5dd2e488","Type":"ContainerStarted","Data":"8859cd69c2564f4d478c7d7612f509618e01b2e33e91df0b9d7359ccc4883287"}
Nov 03 03:13:28.668044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:28.667987    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw" podStartSLOduration=2.230928379 podStartE2EDuration="3.667973371s" totalImagesPullingTime="1.437044992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:28.667697823 +0000 UTC m=+699.898590240" watchObservedRunningTime="2026-11-03 03:13:28.667973371 +0000 UTC m=+699.898865788"
Nov 03 03:13:29.435931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:29.435893    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:29.473193 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:29.473165    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:13:30.041257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.041222    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"]
Nov 03 03:13:30.044941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.044911    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.132766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.132734    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:30.141071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.141045    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.141192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.141077    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7cgc\" (UniqueName: \"kubernetes.io/projected/79d93c40-3f2d-41eb-8006-2493304b0e38-kube-api-access-j7cgc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.141192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.141180    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.141280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.141197    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d93c40-3f2d-41eb-8006-2493304b0e38-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.141280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.141229    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.141280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.141245    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.149443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.149424    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 03:13:30.150542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.150524    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"]
Nov 03 03:13:30.242592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242531    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.242752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242603    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d93c40-3f2d-41eb-8006-2493304b0e38-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.242752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242653    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.242839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242770    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.242899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242844    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.242899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242863    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7cgc\" (UniqueName: \"kubernetes.io/projected/79d93c40-3f2d-41eb-8006-2493304b0e38-kube-api-access-j7cgc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.243005 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242963    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.243005 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.242975    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.243200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.243174    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.245065 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.245037    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d93c40-3f2d-41eb-8006-2493304b0e38-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.245247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.245229    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d93c40-3f2d-41eb-8006-2493304b0e38-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.305453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.305383    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7cgc\" (UniqueName: \"kubernetes.io/projected/79d93c40-3f2d-41eb-8006-2493304b0e38-kube-api-access-j7cgc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-jpqzk\" (UID: \"79d93c40-3f2d-41eb-8006-2493304b0e38\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.358878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.358843    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:30.730298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:30.730272    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"]
Nov 03 03:13:30.732490 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:13:30.732462    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d93c40_3f2d_41eb_8006_2493304b0e38.slice/crio-9f6295e3a0d9a34efb54b6f19ba48726f1766c3dfddf1cb95ffa73dbd4cddffa WatchSource:0}: Error finding container 9f6295e3a0d9a34efb54b6f19ba48726f1766c3dfddf1cb95ffa73dbd4cddffa: Status 404 returned error can't find the container with id 9f6295e3a0d9a34efb54b6f19ba48726f1766c3dfddf1cb95ffa73dbd4cddffa
Nov 03 03:13:31.563648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:31.563616    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk" event={"ID":"79d93c40-3f2d-41eb-8006-2493304b0e38","Type":"ContainerStarted","Data":"05f55ff7ca2492efc80b18cb11968e8b24626495b0ca7a50dc24afaa84081d46"}
Nov 03 03:13:31.564045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:31.563665    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk" event={"ID":"79d93c40-3f2d-41eb-8006-2493304b0e38","Type":"ContainerStarted","Data":"9f6295e3a0d9a34efb54b6f19ba48726f1766c3dfddf1cb95ffa73dbd4cddffa"}
Nov 03 03:13:31.683718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:31.683666    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk" podStartSLOduration=2.098080357 podStartE2EDuration="2.683650853s" totalImagesPullingTime="585.570496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:31.683397953 +0000 UTC m=+702.914290371" watchObservedRunningTime="2026-11-03 03:13:31.683650853 +0000 UTC m=+702.914543291"
Nov 03 03:13:32.359977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:32.359943    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:34.953065 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:34.953026    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:36.436694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:36.436666    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:36.448844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:36.448824    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:36.596617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:36.596587    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-p8clw"
Nov 03 03:13:37.867913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:37.867884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:38.369357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:38.369326    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:13:40.359918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:40.359879    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:40.372015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:40.371991    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:40.614824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:40.614736    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-jpqzk"
Nov 03 03:13:41.900961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.900930    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d45dp"]
Nov 03 03:13:41.904657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.904635    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.945906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.945870    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-home\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.946076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.945924    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.946076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.946032    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.946173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.946101    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.946173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.946129    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjgbh\" (UniqueName: \"kubernetes.io/projected/ef1853a0-c729-463f-ade2-76de6dddca26-kube-api-access-rjgbh\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.946173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.946166    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1853a0-c729-463f-ade2-76de6dddca26-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:41.958435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.958408    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 03:13:41.991885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:41.991844    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:42.010185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.010156    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d45dp"]
Nov 03 03:13:42.047639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.047604    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.047639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.047642    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.047856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.047771    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.047856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.047805    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgbh\" (UniqueName: \"kubernetes.io/projected/ef1853a0-c729-463f-ade2-76de6dddca26-kube-api-access-rjgbh\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.047973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.047858    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1853a0-c729-463f-ade2-76de6dddca26-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.047973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.047916    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-home\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.048112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.048085    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.048176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.048160    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.048277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.048257    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-home\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.050111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.050089    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef1853a0-c729-463f-ade2-76de6dddca26-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.050510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.050492    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef1853a0-c729-463f-ade2-76de6dddca26-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.111861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.111834    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgbh\" (UniqueName: \"kubernetes.io/projected/ef1853a0-c729-463f-ade2-76de6dddca26-kube-api-access-rjgbh\") pod \"echo-simulator-kserve-64b6f8f96c-d45dp\" (UID: \"ef1853a0-c729-463f-ade2-76de6dddca26\") " pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.217139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.217045    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:42.593477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.593341    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-d45dp"]
Nov 03 03:13:42.596135 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:13:42.596109    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1853a0_c729_463f_ade2_76de6dddca26.slice/crio-8b1cd0c12ecc4596d463f57b2fb0a5fb51b352cbf6f2f7fa3dc2d5b5541bbd7c WatchSource:0}: Error finding container 8b1cd0c12ecc4596d463f57b2fb0a5fb51b352cbf6f2f7fa3dc2d5b5541bbd7c: Status 404 returned error can't find the container with id 8b1cd0c12ecc4596d463f57b2fb0a5fb51b352cbf6f2f7fa3dc2d5b5541bbd7c
Nov 03 03:13:42.597419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.597403    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:13:42.612750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:42.612723    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp" event={"ID":"ef1853a0-c729-463f-ade2-76de6dddca26","Type":"ContainerStarted","Data":"8b1cd0c12ecc4596d463f57b2fb0a5fb51b352cbf6f2f7fa3dc2d5b5541bbd7c"}
Nov 03 03:13:45.630196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:45.630159    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp" event={"ID":"ef1853a0-c729-463f-ade2-76de6dddca26","Type":"ContainerStarted","Data":"560ddfc036dffa9cdb345e50b2b59bacdb64d87638185cd31ab5558a37d7ae5a"}
Nov 03 03:13:45.746918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:45.746886    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:45.751165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:45.751121    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp" podStartSLOduration=2.393806649 podStartE2EDuration="4.751109234s" totalImagesPullingTime="2.357302585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:45.74976557 +0000 UTC m=+716.980657999" watchObservedRunningTime="2026-11-03 03:13:45.751109234 +0000 UTC m=+716.982001676"
Nov 03 03:13:46.217872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:46.217837    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:49.110087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.110053    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"]
Nov 03 03:13:49.113583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.113540    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.215036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.215000    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcg9z\" (UniqueName: \"kubernetes.io/projected/e1aa2223-e7d4-4647-8be5-402eb71fed01-kube-api-access-qcg9z\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.215036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.215041    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.215246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.215171    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2223-e7d4-4647-8be5-402eb71fed01-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.215246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.215199    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.215246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.215218    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.215375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.215352    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.217750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.217731    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 03:13:49.219079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.219060    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"]
Nov 03 03:13:49.226059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.226035    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:49.315961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.315925    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2223-e7d4-4647-8be5-402eb71fed01-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.315961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.315968    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.315989    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.316035    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.316058    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcg9z\" (UniqueName: \"kubernetes.io/projected/e1aa2223-e7d4-4647-8be5-402eb71fed01-kube-api-access-qcg9z\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.316075    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.316581    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.316628    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.316769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.316693    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.318786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.318766    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1aa2223-e7d4-4647-8be5-402eb71fed01-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.319181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.319164    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1aa2223-e7d4-4647-8be5-402eb71fed01-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.378530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.378462    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcg9z\" (UniqueName: \"kubernetes.io/projected/e1aa2223-e7d4-4647-8be5-402eb71fed01-kube-api-access-qcg9z\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw\" (UID: \"e1aa2223-e7d4-4647-8be5-402eb71fed01\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.424628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.424594    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:49.798256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:49.798206    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"]
Nov 03 03:13:49.800385 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:13:49.800359    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1aa2223_e7d4_4647_8be5_402eb71fed01.slice/crio-bb3ac7594443470133634860431cccce9d8cd50f64fa46b2be4da4fcf6dcdd26 WatchSource:0}: Error finding container bb3ac7594443470133634860431cccce9d8cd50f64fa46b2be4da4fcf6dcdd26: Status 404 returned error can't find the container with id bb3ac7594443470133634860431cccce9d8cd50f64fa46b2be4da4fcf6dcdd26
Nov 03 03:13:50.653379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:50.653295    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" event={"ID":"e1aa2223-e7d4-4647-8be5-402eb71fed01","Type":"ContainerStarted","Data":"41e6205efbdcaa8619e085e646ce77076f2e36cf1cda6d2813eff9f9d44e643b"}
Nov 03 03:13:50.653379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:50.653336    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" event={"ID":"e1aa2223-e7d4-4647-8be5-402eb71fed01","Type":"ContainerStarted","Data":"bb3ac7594443470133634860431cccce9d8cd50f64fa46b2be4da4fcf6dcdd26"}
Nov 03 03:13:50.774437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:50.774379    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podStartSLOduration=1.194044374 podStartE2EDuration="1.774366734s" totalImagesPullingTime="580.32236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:50.773507319 +0000 UTC m=+722.004399736" watchObservedRunningTime="2026-11-03 03:13:50.774366734 +0000 UTC m=+722.005259152"
Nov 03 03:13:50.983670 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:13:50.983580    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:13:51.425605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:51.425570    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:51.535350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:51.535314    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:13:52.217478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:52.217446    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:52.230025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:52.230003    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:52.673211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:52.673182    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp"
Nov 03 03:13:54.090422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:54.090379    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:13:59.425612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:59.425576    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:59.437766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:59.437746    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:13:59.705757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:13:59.705682    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:14:01.170233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.170199    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"]
Nov 03 03:14:01.173659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.173642    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.227341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.227307    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gfd7r\""
Nov 03 03:14:01.279425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.279397    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"]
Nov 03 03:14:01.324983 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.324950    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.324992    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hdn\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325027    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325141    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325192    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325287    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325312    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325338    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.325395    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.426805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.426714    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.426805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.426758    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hdn\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.426896    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.426939    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.426963    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427004    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427035    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427078    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427120    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427333    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427402    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427506    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427531    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.427691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.427637    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.429564 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.429518    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.429803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.429785    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.489347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.489314    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.489487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.489439    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hdn\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn\") pod \"maas-default-gateway-openshift-default-76f4876667-42kt8\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:01.785912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:01.785826    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:02.264396 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:02.264366    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"]
Nov 03 03:14:02.266903 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:14:02.266876    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a23be2_6f70_4b11_88fe_443fcb40f732.slice/crio-0f665192118ecedcd25b83383ac1fed901af1b9bc54bf3319bc2ab42bd40a5ae WatchSource:0}: Error finding container 0f665192118ecedcd25b83383ac1fed901af1b9bc54bf3319bc2ab42bd40a5ae: Status 404 returned error can't find the container with id 0f665192118ecedcd25b83383ac1fed901af1b9bc54bf3319bc2ab42bd40a5ae
Nov 03 03:14:02.710410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:02.710376    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" event={"ID":"79a23be2-6f70-4b11-88fe-443fcb40f732","Type":"ContainerStarted","Data":"0f665192118ecedcd25b83383ac1fed901af1b9bc54bf3319bc2ab42bd40a5ae"}
Nov 03 03:14:07.707754 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:07.707718    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:14:07.708154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:07.707805    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:14:07.708154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:07.707855    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:14:08.738442 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:08.738408    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" event={"ID":"79a23be2-6f70-4b11-88fe-443fcb40f732","Type":"ContainerStarted","Data":"7278c0bf23244f406a6119f026237f46f1d6201c971a1fd22f09776bd1139277"}
Nov 03 03:14:08.786741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:08.786679    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:08.788537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:08.788504    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:08.788680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:08.788575    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:08.950784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:08.950719    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podStartSLOduration=2.512613376 podStartE2EDuration="7.95070348s" totalImagesPullingTime="5.438090104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:08.950253185 +0000 UTC m=+740.181145602" watchObservedRunningTime="2026-11-03 03:14:08.95070348 +0000 UTC m=+740.181595970"
Nov 03 03:14:09.787036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:09.786993    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:09.787463 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:09.787054    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:10.786961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:10.786924    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:10.787115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:10.786979    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:11.786897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:11.786857    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:11.787211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:11.786938    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:12.786952 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:12.786918    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:12.787141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:12.786976    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:13.786204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:13.786171    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:13.786621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:13.786221    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:14.786610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:14.786567    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-42kt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 03:14:14.786992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:14.786647    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 03:14:16.078530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:16.078496    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:16.079029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:16.078972    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:16.080129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:16.080098    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:14:19.798932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:19.798897    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:19.949643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:19.949605    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:20.396159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:20.396128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:20.548832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:20.548798    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:20.984346 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:14:20.984313    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:14:22.165364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:22.165327    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:22.318120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:22.318085    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:31.433231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.433196    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"]
Nov 03 03:14:31.436691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.436673    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503104    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503139    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503250    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7f2\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503298    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503324    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503469    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503512    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503599    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.503649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.503625    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.542273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.542230    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"]
Nov 03 03:14:31.604439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604404    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7f2\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604461    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604486    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604567    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604595    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604625    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604647    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604680    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.604921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604707    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.605069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.604928    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.605301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.605277    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.605401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.605309    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.605401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.605264    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.605401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.605363    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.607798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.607769    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.607947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.607930    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.666742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.666709    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.666992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.666966    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7f2\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2\") pod \"maas-default-gateway-openshift-default-76f4876667-4t4rg\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:31.749426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:31.749342    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:32.177814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:32.177788    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"]
Nov 03 03:14:32.179747 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:14:32.179727    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b5226a_d141_496c_8a53_1a29728f7590.slice/crio-ca8b9262e0f915c8730f827223defb6a1db0551458aed057f799ab3eb690bd95 WatchSource:0}: Error finding container ca8b9262e0f915c8730f827223defb6a1db0551458aed057f799ab3eb690bd95: Status 404 returned error can't find the container with id ca8b9262e0f915c8730f827223defb6a1db0551458aed057f799ab3eb690bd95
Nov 03 03:14:32.181810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:32.181779    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:14:32.181872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:32.181845    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:14:32.181918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:32.181873    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:14:32.845933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:32.845901    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" event={"ID":"a2b5226a-d141-496c-8a53-1a29728f7590","Type":"ContainerStarted","Data":"b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823"}
Nov 03 03:14:32.846425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:32.845959    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" event={"ID":"a2b5226a-d141-496c-8a53-1a29728f7590","Type":"ContainerStarted","Data":"ca8b9262e0f915c8730f827223defb6a1db0551458aed057f799ab3eb690bd95"}
Nov 03 03:14:33.021850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:33.021783    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podStartSLOduration=2.021764053 podStartE2EDuration="2.021764053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:33.019999444 +0000 UTC m=+764.250891863" watchObservedRunningTime="2026-11-03 03:14:33.021764053 +0000 UTC m=+764.252656475"
Nov 03 03:14:33.749649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:33.749611    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:33.751050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:33.751023    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 03:14:33.751169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:33.751075    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 03:14:34.749861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:34.749826    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 03:14:34.750250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:34.749881    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 03:14:35.749777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:35.749745    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 03:14:35.749962 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:35.749796    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 03:14:36.253674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:36.253637    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:36.404835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:36.404805    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:36.750506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:36.750468    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 03:14:36.750917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:36.750543    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 03:14:37.750712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:37.750678    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 03:14:37.751123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:37.750734    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 03:14:38.749845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:38.749811    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 03:14:38.750017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:38.749866    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 03:14:40.491157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:40.491110    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:40.491589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:40.491474    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:40.492254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:40.492233    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 03:14:47.963680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:47.963642    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:48.110750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:48.110707    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:50.982570 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:14:50.982524    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:14:51.886405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:51.886367    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:58.381880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:58.381842    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:14:59.428827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.428796    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"]
Nov 03 03:14:59.432249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.432230    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482505 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482462    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482527    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482597    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482783    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482832    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482895    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqdf\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.482960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482925    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.483060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.482965    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.483060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.483002    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.530303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.530270    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"]
Nov 03 03:14:59.558957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.558921    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b4ff4779-openshift-default-dockercfg-6h5c4\""
Nov 03 03:14:59.584501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584465    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584505    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584523    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584591    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584613    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584755    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqdf\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584790    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584831    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584878    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.584965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584932    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.585164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.584983    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.585164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.585025    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.585164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.585076    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.585629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.585522    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.587371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.587345    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.587466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.587451    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.649358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.649318    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.649358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.649357    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqdf\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf\") pod \"e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") " pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:14:59.745196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:14:59.745116    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:00.195895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.195866    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"]
Nov 03 03:15:00.197658 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:00.197629    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199ab1e7_67f3_485c_b25c_fe8e088e754f.slice/crio-c4639ec7beffd40c491070282682b0199c6ab4e321c0945a385a72e924552d47 WatchSource:0}: Error finding container c4639ec7beffd40c491070282682b0199c6ab4e321c0945a385a72e924552d47: Status 404 returned error can't find the container with id c4639ec7beffd40c491070282682b0199c6ab4e321c0945a385a72e924552d47
Nov 03 03:15:00.199968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.199935    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:00.200048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.200003    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:00.200048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.200032    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:00.340378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.340347    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:00.994340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.994305    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" event={"ID":"199ab1e7-67f3-485c-b25c-fe8e088e754f","Type":"ContainerStarted","Data":"89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"}
Nov 03 03:15:00.994788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:00.994372    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" event={"ID":"199ab1e7-67f3-485c-b25c-fe8e088e754f","Type":"ContainerStarted","Data":"c4639ec7beffd40c491070282682b0199c6ab4e321c0945a385a72e924552d47"}
Nov 03 03:15:01.119323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.119272    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podStartSLOduration=2.11925673 podStartE2EDuration="2.11925673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:01.117367078 +0000 UTC m=+792.348259497" watchObservedRunningTime="2026-11-03 03:15:01.11925673 +0000 UTC m=+792.350149218"
Nov 03 03:15:01.560635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.560594    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"]
Nov 03 03:15:01.564328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.564302    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608077    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608194    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608233    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608288    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608408    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608441    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608485    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608616    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.608845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.608640    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995vv\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.617924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.617895    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-56094c67-openshift-default-dockercfg-jj2lr\""
Nov 03 03:15:01.633876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.633837    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"]
Nov 03 03:15:01.652781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.652753    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:01.710096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710056    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710108    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995vv\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710184    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710231    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710267    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710329    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710356    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710381    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710408    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710775    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710815    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710858    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.710912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.710865    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.711181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.711029    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.712850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.712829    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.712961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.712942    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.745281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.745248    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:01.750387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.750360    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:01.773647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.773615    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.783818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.774921    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-995vv\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv\") pod \"e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") " pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:01.804091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.804055    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:01.878301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:01.878259    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:02.001028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:02.000997    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:02.002022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:02.002003    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:02.307497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:02.307472    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"]
Nov 03 03:15:02.309358 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:02.309330    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9949878_41c3_44a4_9e5e_fc7430c43e4c.slice/crio-330109b142d1e9c98c2569e5276a3a3b9392906116c1f9ed1b6554640f015de0 WatchSource:0}: Error finding container 330109b142d1e9c98c2569e5276a3a3b9392906116c1f9ed1b6554640f015de0: Status 404 returned error can't find the container with id 330109b142d1e9c98c2569e5276a3a3b9392906116c1f9ed1b6554640f015de0
Nov 03 03:15:02.311715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:02.311681    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:02.311793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:02.311754    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:02.311793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:02.311785    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:03.007982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.007943    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" event={"ID":"d9949878-41c3-44a4-9e5e-fc7430c43e4c","Type":"ContainerStarted","Data":"d02dd1b3f20c9c6c18afdb8014dccb13f0728055345c9b6a05c2eefc4fa29a49"}
Nov 03 03:15:03.008420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.008040    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" event={"ID":"d9949878-41c3-44a4-9e5e-fc7430c43e4c","Type":"ContainerStarted","Data":"330109b142d1e9c98c2569e5276a3a3b9392906116c1f9ed1b6554640f015de0"}
Nov 03 03:15:03.142880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.142824    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podStartSLOduration=2.142808636 podStartE2EDuration="2.142808636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:03.1415795 +0000 UTC m=+794.372471912" watchObservedRunningTime="2026-11-03 03:15:03.142808636 +0000 UTC m=+794.373701053"
Nov 03 03:15:03.179783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.179744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:03.491042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.491005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:03.670576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.670509    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:03.840202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.840118    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:03.878630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.878601    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:03.883925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:03.883903    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:04.009878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:04.009846    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:04.013405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:04.013379    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:04.014077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:04.014062    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:04.179783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:04.179752    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:04.936117 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:04.936071    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:05.155237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:05.155203    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:05.512058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:05.512027    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:05.662960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:05.662924    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:06.092658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:06.092628    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:06.244220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:06.244186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:06.436923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:06.436889    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:07.356635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.356602    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:07.557298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.557265    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:07.707843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.707815    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:07.854389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.854353    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"]
Nov 03 03:15:07.858103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.858086    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:07.916725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.916695    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"]
Nov 03 03:15:07.974211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.974117    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:07.974382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.974251    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q28d\" (UniqueName: \"kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d\") pod \"payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:07.975025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.975005    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4fb905b5-dockercfg-t9mbj\""
Nov 03 03:15:07.975025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:07.975007    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4fb905b5\""
Nov 03 03:15:08.027922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.027889    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"]
Nov 03 03:15:08.031518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.031491    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.075000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.074909    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") " pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.075205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.075058    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q28d\" (UniqueName: \"kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d\") pod \"payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:08.075205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.075179    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:08.075284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.075251    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczlv\" (UniqueName: \"kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv\") pod \"payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") " pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.075931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.075912    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:08.090491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.090456    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"]
Nov 03 03:15:08.137384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.137353    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q28d\" (UniqueName: \"kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d\") pod \"payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:08.171455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.171418    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:08.175786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.175760    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczlv\" (UniqueName: \"kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv\") pod \"payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") " pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.175888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.175804    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") " pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.176504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.176484    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") " pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.243263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.243230    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczlv\" (UniqueName: \"kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv\") pod \"payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") " pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.345446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.345398    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:08.547363 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.547309    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"]
Nov 03 03:15:08.549637 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:08.549611    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f598cb_0ac3_4468_af24_0e51f2a7a7e2.slice/crio-68ff10de300388e988975f3819a04d46d8d03f81123c2e6e9d77df9e7ce4591f WatchSource:0}: Error finding container 68ff10de300388e988975f3819a04d46d8d03f81123c2e6e9d77df9e7ce4591f: Status 404 returned error can't find the container with id 68ff10de300388e988975f3819a04d46d8d03f81123c2e6e9d77df9e7ce4591f
Nov 03 03:15:08.721015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:08.720987    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"]
Nov 03 03:15:08.722809 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:08.722780    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4474c9b2_6bac_4ef4_a223_0622cc17e499.slice/crio-767f95c53f532e483d519f812c59c738400d8e851170e0aac688645d6b71b7bb WatchSource:0}: Error finding container 767f95c53f532e483d519f812c59c738400d8e851170e0aac688645d6b71b7bb: Status 404 returned error can't find the container with id 767f95c53f532e483d519f812c59c738400d8e851170e0aac688645d6b71b7bb
Nov 03 03:15:09.039636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.039595    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg" event={"ID":"4474c9b2-6bac-4ef4-a223-0622cc17e499","Type":"ContainerStarted","Data":"ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663"}
Nov 03 03:15:09.039847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.039661    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg" event={"ID":"4474c9b2-6bac-4ef4-a223-0622cc17e499","Type":"ContainerStarted","Data":"767f95c53f532e483d519f812c59c738400d8e851170e0aac688645d6b71b7bb"}
Nov 03 03:15:09.039847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.039776    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:09.041154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.041121    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z" event={"ID":"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2","Type":"ContainerStarted","Data":"616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96"}
Nov 03 03:15:09.041251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.041171    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:09.041251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.041187    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z" event={"ID":"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2","Type":"ContainerStarted","Data":"68ff10de300388e988975f3819a04d46d8d03f81123c2e6e9d77df9e7ce4591f"}
Nov 03 03:15:09.159378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.159330    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg" podStartSLOduration=2.159317484 podStartE2EDuration="2.159317484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:09.158671695 +0000 UTC m=+800.389564113" watchObservedRunningTime="2026-11-03 03:15:09.159317484 +0000 UTC m=+800.390209904"
Nov 03 03:15:09.280875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.280802    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z" podStartSLOduration=2.280781307 podStartE2EDuration="2.280781307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:09.278893673 +0000 UTC m=+800.509786091" watchObservedRunningTime="2026-11-03 03:15:09.280781307 +0000 UTC m=+800.511673726"
Nov 03 03:15:09.402579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.402514    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:09.644784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.644742    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:09.798229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.798139    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:09.947252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:09.947218    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:10.105784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:10.105702    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:12.284613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:12.284574    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:12.466500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:12.466465    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:13.695605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:13.695571    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:13.865825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:13.865785    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:14.767718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:14.767682    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:14.948247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:14.948215    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:15.182690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.182663    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"]
Nov 03 03:15:15.186302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.186286    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.229499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.229465    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:15.243571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.243523    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"]
Nov 03 03:15:15.251048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.251013    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pc8\" (UniqueName: \"kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8\") pod \"payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.251198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.251057    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.292168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.292087    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b4ff4779-dockercfg-n5gjj\""
Nov 03 03:15:15.292168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.292126    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b4ff4779\""
Nov 03 03:15:15.352063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.352028    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pc8\" (UniqueName: \"kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8\") pod \"payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.352268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.352076    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.352868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.352844    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.388457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.388426    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:15.414337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.414303    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pc8\" (UniqueName: \"kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8\") pod \"payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.499631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.499539    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:15.544307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.544266    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:15.877410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:15.877300    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"]
Nov 03 03:15:15.880055 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:15.880035    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaec0252_144b_4dcb_b5d6_8fc3ddeca35d.slice/crio-bc00fdb72d04546e2d04c087d90dcad1c9221610af6a3e822ca1005ceb4a7a2c WatchSource:0}: Error finding container bc00fdb72d04546e2d04c087d90dcad1c9221610af6a3e822ca1005ceb4a7a2c: Status 404 returned error can't find the container with id bc00fdb72d04546e2d04c087d90dcad1c9221610af6a3e822ca1005ceb4a7a2c
Nov 03 03:15:16.077652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:16.077622    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67" event={"ID":"baec0252-144b-4dcb-b5d6-8fc3ddeca35d","Type":"ContainerStarted","Data":"aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d"}
Nov 03 03:15:16.077814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:16.077668    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67" event={"ID":"baec0252-144b-4dcb-b5d6-8fc3ddeca35d","Type":"ContainerStarted","Data":"bc00fdb72d04546e2d04c087d90dcad1c9221610af6a3e822ca1005ceb4a7a2c"}
Nov 03 03:15:16.077814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:16.077768    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:16.196736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:16.196625    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67" podStartSLOduration=1.196609214 podStartE2EDuration="1.196609214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:16.195949912 +0000 UTC m=+807.426842329" watchObservedRunningTime="2026-11-03 03:15:16.196609214 +0000 UTC m=+807.427501628"
Nov 03 03:15:17.288447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.288411    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:17.461623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.461590    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:17.688591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.688562    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"]
Nov 03 03:15:17.692160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.692145    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:17.745411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.745376    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-56094c67\""
Nov 03 03:15:17.783230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.783198    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brrq\" (UniqueName: \"kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq\") pod \"payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:17.783431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.783352    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:17.796535 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.796503    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-56094c67-dockercfg-mhz42\""
Nov 03 03:15:17.800484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.800455    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"]
Nov 03 03:15:17.884375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.884342    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:17.884537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.884398    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brrq\" (UniqueName: \"kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq\") pod \"payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:17.885078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.885061    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:17.946846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:17.946780    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brrq\" (UniqueName: \"kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq\") pod \"payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:18.005177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:18.005143    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:18.376449 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:18.376347    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"]
Nov 03 03:15:18.390593 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:18.381098    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44482ad5_99e6_4dce_9679_014b20e66e0a.slice/crio-0e37dc4c4ec56a04efa92d2daa16fd634daa022324e14c9b768d29b8af2f91c7 WatchSource:0}: Error finding container 0e37dc4c4ec56a04efa92d2daa16fd634daa022324e14c9b768d29b8af2f91c7: Status 404 returned error can't find the container with id 0e37dc4c4ec56a04efa92d2daa16fd634daa022324e14c9b768d29b8af2f91c7
Nov 03 03:15:19.096168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:19.096135    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng" event={"ID":"44482ad5-99e6-4dce-9679-014b20e66e0a","Type":"ContainerStarted","Data":"86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b"}
Nov 03 03:15:19.096379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:19.096200    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:19.096379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:19.096224    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng" event={"ID":"44482ad5-99e6-4dce-9679-014b20e66e0a","Type":"ContainerStarted","Data":"0e37dc4c4ec56a04efa92d2daa16fd634daa022324e14c9b768d29b8af2f91c7"}
Nov 03 03:15:19.225506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:19.225451    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng" podStartSLOduration=2.225436657 podStartE2EDuration="2.225436657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:19.223891546 +0000 UTC m=+810.454783963" watchObservedRunningTime="2026-11-03 03:15:19.225436657 +0000 UTC m=+810.456329075"
Nov 03 03:15:20.049332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.049305    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:15:20.049744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.049372    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:15:20.221037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.221005    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"]
Nov 03 03:15:20.225385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.225359    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.277394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.277366    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"]
Nov 03 03:15:20.278937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.278917    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-56094c67-serving-cert\""
Nov 03 03:15:20.311048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.310977    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.311048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.311023    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjk6b\" (UniqueName: \"kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.311225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.311182    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.311271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.311225    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.374592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.374563    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:20.412009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.411969    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.412009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.412008    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.412265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.412067    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.412265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.412094    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjk6b\" (UniqueName: \"kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.412265 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:15:20.412203    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-serving-cert: secret "maas-api-e2e-ait-56094c67-serving-cert" not found
Nov 03 03:15:20.412397 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:15:20.412291    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls podName:8351ee8e-50ff-48df-8f25-c9f709ea2dca nodeName:}" failed. No retries permitted until 2026-11-03 03:15:20.912265371 +0000 UTC m=+812.143157780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls") pod "maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" (UID: "8351ee8e-50ff-48df-8f25-c9f709ea2dca") : secret "maas-api-e2e-ait-56094c67-serving-cert" not found
Nov 03 03:15:20.412832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.412814    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.434307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.434245    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.474771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.474730    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjk6b\" (UniqueName: \"kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.538913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.538855    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:20.687067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.687036    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:20.918708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.918676    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.921236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:20.921206    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\") pod \"maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:20.983949 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:15:20.983856    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:15:21.139651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:21.139617    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:21.515245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:21.515221    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"]
Nov 03 03:15:21.517139 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:21.517108    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8351ee8e_50ff_48df_8f25_c9f709ea2dca.slice/crio-774a518f544ae3e90cefd5113d26fc4d710acb58426b189159ad26abbcae73e8 WatchSource:0}: Error finding container 774a518f544ae3e90cefd5113d26fc4d710acb58426b189159ad26abbcae73e8: Status 404 returned error can't find the container with id 774a518f544ae3e90cefd5113d26fc4d710acb58426b189159ad26abbcae73e8
Nov 03 03:15:21.624079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:21.624050    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:15:22.112048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:22.112009    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" event={"ID":"8351ee8e-50ff-48df-8f25-c9f709ea2dca","Type":"ContainerStarted","Data":"774a518f544ae3e90cefd5113d26fc4d710acb58426b189159ad26abbcae73e8"}
Nov 03 03:15:23.118628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:23.118587    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" event={"ID":"8351ee8e-50ff-48df-8f25-c9f709ea2dca","Type":"ContainerStarted","Data":"ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f"}
Nov 03 03:15:23.119044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:23.118749    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:23.250621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:23.250574    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" podStartSLOduration=2.2213113509999998 podStartE2EDuration="3.250541869s" totalImagesPullingTime="1.029230518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:23.249392121 +0000 UTC m=+814.480284553" watchObservedRunningTime="2026-11-03 03:15:23.250541869 +0000 UTC m=+814.481434287"
Nov 03 03:15:25.561807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:25.561736    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:26.781576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:26.781170    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:26.950985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:26.950945    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:27.104896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.104787    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:15:27.339004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.338970    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"]
Nov 03 03:15:27.412219 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.412184    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"]
Nov 03 03:15:27.412685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.412663    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.485599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.485536    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-69e611c0\""
Nov 03 03:15:27.486072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.486049    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-69e611c0-dockercfg-g974c\""
Nov 03 03:15:27.514247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.514214    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76x6\" (UniqueName: \"kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6\") pod \"payload-processing-e2e-derive-69e611c0-7897b884f-q4htw\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") " pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.514247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.514252    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume\") pod \"payload-processing-e2e-derive-69e611c0-7897b884f-q4htw\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") " pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.616076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.616039    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76x6\" (UniqueName: \"kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6\") pod \"payload-processing-e2e-derive-69e611c0-7897b884f-q4htw\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") " pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.616076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.616076    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume\") pod \"payload-processing-e2e-derive-69e611c0-7897b884f-q4htw\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") " pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.616806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.616785    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume\") pod \"payload-processing-e2e-derive-69e611c0-7897b884f-q4htw\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") " pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.678930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.678861    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76x6\" (UniqueName: \"kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6\") pod \"payload-processing-e2e-derive-69e611c0-7897b884f-q4htw\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") " pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:27.727867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:27.727838    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:28.101944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:28.101915    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"]
Nov 03 03:15:28.103971 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:28.103944    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1399fba_8170_4593_9916_432e5ca21223.slice/crio-a9faef9e38a3a13ba2ec3e092f7b148627fe91db7c865b5c49f56cff0d81628e WatchSource:0}: Error finding container a9faef9e38a3a13ba2ec3e092f7b148627fe91db7c865b5c49f56cff0d81628e: Status 404 returned error can't find the container with id a9faef9e38a3a13ba2ec3e092f7b148627fe91db7c865b5c49f56cff0d81628e
Nov 03 03:15:28.145479 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:28.145441    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw" event={"ID":"b1399fba-8170-4593-9916-432e5ca21223","Type":"ContainerStarted","Data":"a9faef9e38a3a13ba2ec3e092f7b148627fe91db7c865b5c49f56cff0d81628e"}
Nov 03 03:15:29.129197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:29.129167    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:15:29.153099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:29.153063    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw" event={"ID":"b1399fba-8170-4593-9916-432e5ca21223","Type":"ContainerStarted","Data":"3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684"}
Nov 03 03:15:29.153272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:29.153195    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:29.374284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:29.374234    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw" podStartSLOduration=2.374219348 podStartE2EDuration="2.374219348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:29.373458109 +0000 UTC m=+820.604350552" watchObservedRunningTime="2026-11-03 03:15:29.374219348 +0000 UTC m=+820.605111767"
Nov 03 03:15:30.103984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:30.103954    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:15:30.389434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:30.389393    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:30.556247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:30.556204    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:30.705885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:30.705807    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:32.002840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.002807    5515 patch_prober.go:29] interesting pod/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 03:15:32.003248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.002867    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 03:15:32.192690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.192653    5515 generic.go:356] "Generic (PLEG): container finished" podID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerID="89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea" exitCode=0
Nov 03 03:15:32.192866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.192731    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" event={"ID":"199ab1e7-67f3-485c-b25c-fe8e088e754f","Type":"ContainerDied","Data":"89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"}
Nov 03 03:15:32.193120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.193107    5515 scope.go:122] "RemoveContainer" containerID="89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"
Nov 03 03:15:32.194314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.194286    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:32.194389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.194356    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:32.194424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:32.194386    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:33.217600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:33.217539    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" event={"ID":"199ab1e7-67f3-485c-b25c-fe8e088e754f","Type":"ContainerStarted","Data":"caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20"}
Nov 03 03:15:33.746145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:33.746120    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:33.746424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:33.746362    5515 patch_prober.go:29] interesting pod/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 03:15:33.746585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:33.746544    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 03:15:34.746162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:34.746112    5515 patch_prober.go:29] interesting pod/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 03:15:34.746610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:34.746177    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 03:15:35.190463 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.190427    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"]
Nov 03 03:15:35.206451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.206423    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.233127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.233098    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s89j\" (UniqueName: \"kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.233389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.233373    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.233523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.233500    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.233627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.233593    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.247279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.247239    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"]
Nov 03 03:15:35.300748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.300705    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-69e611c0-serving-cert\""
Nov 03 03:15:35.335031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.334999    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.335195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.335046    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s89j\" (UniqueName: \"kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.335195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.335118    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.335195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.335134    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.335195 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:15:35.335179    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-serving-cert: secret "maas-api-e2e-derive-69e611c0-serving-cert" not found
Nov 03 03:15:35.335331 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:15:35.335260    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls podName:ddd84fab-f048-41d1-b34d-11d27184f6c0 nodeName:}" failed. No retries permitted until 2026-11-03 03:15:35.835235843 +0000 UTC m=+827.066128258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls") pod "maas-api-e2e-derive-69e611c0-dd586fc8-8466v" (UID: "ddd84fab-f048-41d1-b34d-11d27184f6c0") : secret "maas-api-e2e-derive-69e611c0-serving-cert" not found
Nov 03 03:15:35.336266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.336240    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.340584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.337776    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.406574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.406393    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:35.414943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.414917    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s89j\" (UniqueName: \"kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.555453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.555372    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:35.707537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.707497    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:35.746567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.746518    5515 patch_prober.go:29] interesting pod/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 03:15:35.746945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.746613    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 03:15:35.841592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.841482    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:35.844167 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:35.844144    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\") pod \"maas-api-e2e-derive-69e611c0-dd586fc8-8466v\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:36.126403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:36.126370    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:36.532912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:36.532882    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"]
Nov 03 03:15:36.534524 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:15:36.534491    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd84fab_f048_41d1_b34d_11d27184f6c0.slice/crio-41b9e9b2bd1c854c07072573b381d0ddf565ec406d307f08f728333232ad87b2 WatchSource:0}: Error finding container 41b9e9b2bd1c854c07072573b381d0ddf565ec406d307f08f728333232ad87b2: Status 404 returned error can't find the container with id 41b9e9b2bd1c854c07072573b381d0ddf565ec406d307f08f728333232ad87b2
Nov 03 03:15:36.745913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:36.745874    5515 patch_prober.go:29] interesting pod/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 03:15:36.746115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:36.745932    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 03:15:37.242364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:37.242326    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v" event={"ID":"ddd84fab-f048-41d1-b34d-11d27184f6c0","Type":"ContainerStarted","Data":"41b9e9b2bd1c854c07072573b381d0ddf565ec406d307f08f728333232ad87b2"}
Nov 03 03:15:37.745999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:37.745961    5515 patch_prober.go:29] interesting pod/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 03:15:37.746192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:37.746027    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 03:15:38.250331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:38.250293    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v" event={"ID":"ddd84fab-f048-41d1-b34d-11d27184f6c0","Type":"ContainerStarted","Data":"91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a"}
Nov 03 03:15:38.250774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:38.250428    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:38.372155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:38.372091    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v" podStartSLOduration=2.573651676 podStartE2EDuration="3.372077851s" totalImagesPullingTime="798.426175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:38.370931837 +0000 UTC m=+829.601824267" watchObservedRunningTime="2026-11-03 03:15:38.372077851 +0000 UTC m=+829.602970268"
Nov 03 03:15:38.750897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:38.750866    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:38.751266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:38.751241    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:38.751914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:38.751897    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:15:39.802271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:39.802227    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:15:40.161009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.160982    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:15:40.262065 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.262029    5515 generic.go:356] "Generic (PLEG): container finished" podID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerID="d02dd1b3f20c9c6c18afdb8014dccb13f0728055345c9b6a05c2eefc4fa29a49" exitCode=0
Nov 03 03:15:40.262250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.262114    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" event={"ID":"d9949878-41c3-44a4-9e5e-fc7430c43e4c","Type":"ContainerDied","Data":"d02dd1b3f20c9c6c18afdb8014dccb13f0728055345c9b6a05c2eefc4fa29a49"}
Nov 03 03:15:40.262567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.262536    5515 scope.go:122] "RemoveContainer" containerID="d02dd1b3f20c9c6c18afdb8014dccb13f0728055345c9b6a05c2eefc4fa29a49"
Nov 03 03:15:40.263814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.263789    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:40.263899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.263852    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:40.263899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.263882    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:15:40.477807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:40.477700    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:41.291250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:41.291215    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" event={"ID":"d9949878-41c3-44a4-9e5e-fc7430c43e4c","Type":"ContainerStarted","Data":"fc55d2db795452d655f98295ba910b9f25f054768e6818b767b6bf7768e1e66f"}
Nov 03 03:15:41.882573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:41.881936    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:41.882573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:41.882067    5515 patch_prober.go:29] interesting pod/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 03:15:41.882573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:41.882113    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 03:15:42.882017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:42.881977    5515 patch_prober.go:29] interesting pod/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 03:15:42.882652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:42.882618    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 03:15:43.882077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:43.882033    5515 patch_prober.go:29] interesting pod/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 03:15:43.882736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:43.882704    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 03:15:44.275628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:44.261664    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:15:44.880857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:44.880822    5515 patch_prober.go:29] interesting pod/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 03:15:44.881202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:44.881171    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 03:15:45.339252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:45.339177    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:45.879973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:45.879936    5515 patch_prober.go:29] interesting pod/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 03:15:45.880385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:45.880355    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 03:15:46.883031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:46.882999    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:46.883441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:46.883359    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:46.883939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:46.883919    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:15:48.391109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:48.391070    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:15:50.336920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:50.336885    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:50.984522 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:15:50.984486    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:15:52.234058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:52.234007    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:52.376910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:52.376874    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:15:55.757901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:15:55.757865    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:01.084688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:01.084648    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:03.647039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:03.646999    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"]
Nov 03 03:16:03.888216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:03.888186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:04.034914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:04.034830    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:06.421278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:06.421252    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:08.421009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:08.420968    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"]
Nov 03 03:16:08.530869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:08.530840    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"]
Nov 03 03:16:08.647404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:08.647356    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67" podUID="baec0252-144b-4dcb-b5d6-8fc3ddeca35d" containerName="payload-pre-processing" containerID="cri-o://aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d" gracePeriod=25
Nov 03 03:16:08.781248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:08.781211    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:08.910432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:08.910406    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:16:08.929043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:08.929013    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:09.053482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.053393    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8\" (UniqueName: \"kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8\") pod \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") "
Nov 03 03:16:09.053661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.053574    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume\") pod \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\" (UID: \"baec0252-144b-4dcb-b5d6-8fc3ddeca35d\") "
Nov 03 03:16:09.053975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.053952    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume" pod "baec0252-144b-4dcb-b5d6-8fc3ddeca35d" (UID: "baec0252-144b-4dcb-b5d6-8fc3ddeca35d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:09.055828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.055796    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8" pod "baec0252-144b-4dcb-b5d6-8fc3ddeca35d" (UID: "baec0252-144b-4dcb-b5d6-8fc3ddeca35d"). InnerVolumeSpecName "kube-api-access-p4pc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:09.154900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.154864    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:09.154900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.154895    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pc8\" (UniqueName: \"kubernetes.io/projected/baec0252-144b-4dcb-b5d6-8fc3ddeca35d-kube-api-access-p4pc8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:09.438739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.438703    5515 generic.go:356] "Generic (PLEG): container finished" podID="baec0252-144b-4dcb-b5d6-8fc3ddeca35d" containerID="aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d" exitCode=0
Nov 03 03:16:09.439159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.438753    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"
Nov 03 03:16:09.439159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.438784    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67" event={"ID":"baec0252-144b-4dcb-b5d6-8fc3ddeca35d","Type":"ContainerDied","Data":"aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d"}
Nov 03 03:16:09.439159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.438833    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67" event={"ID":"baec0252-144b-4dcb-b5d6-8fc3ddeca35d","Type":"ContainerDied","Data":"bc00fdb72d04546e2d04c087d90dcad1c9221610af6a3e822ca1005ceb4a7a2c"}
Nov 03 03:16:09.439159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.438853    5515 scope.go:122] "RemoveContainer" containerID="aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d"
Nov 03 03:16:09.447895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.447876    5515 scope.go:122] "RemoveContainer" containerID="aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d"
Nov 03 03:16:09.448191 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:09.448168    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d\": container with ID starting with aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d not found: ID does not exist" containerID="aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d"
Nov 03 03:16:09.448286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.448197    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d"} err="failed to get container status \"aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d\": rpc error: code = NotFound desc = could not find container \"aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d\": container with ID starting with aa12363dabdb0c62624d7fac47f3fb72b3299abb69c136cf63c9cbe616ce450d not found: ID does not exist"
Nov 03 03:16:09.606399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.606366    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"]
Nov 03 03:16:09.610296 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:09.610272    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4ff4779-7588557899-64l67"]
Nov 03 03:16:10.892873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:10.892842    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baec0252-144b-4dcb-b5d6-8fc3ddeca35d" path="/var/lib/kubelet/pods/baec0252-144b-4dcb-b5d6-8fc3ddeca35d/volumes"
Nov 03 03:16:11.095661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:11.095622    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:12.383862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:12.383811    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:12.867752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:12.867721    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:13.158434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.158352    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:13.421371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.421269    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg" podUID="4474c9b2-6bac-4ef4-a223-0622cc17e499" containerName="payload-processing" containerID="cri-o://ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663" gracePeriod=25
Nov 03 03:16:13.491216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.491183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:13.500804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.500771    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"]
Nov 03 03:16:13.501003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.500983    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy" containerID="cri-o://caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20" gracePeriod=30
Nov 03 03:16:13.531120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.531057    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z" podUID="b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" containerName="payload-pre-processing" containerID="cri-o://616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96" gracePeriod=25
Nov 03 03:16:13.723486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.723457    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:16:13.798993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.798969    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:16:13.810199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.810174    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv\" (UniqueName: \"kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv\") pod \"4474c9b2-6bac-4ef4-a223-0622cc17e499\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") "
Nov 03 03:16:13.810338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.810220    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume\") pod \"4474c9b2-6bac-4ef4-a223-0622cc17e499\" (UID: \"4474c9b2-6bac-4ef4-a223-0622cc17e499\") "
Nov 03 03:16:13.810724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.810695    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume" pod "4474c9b2-6bac-4ef4-a223-0622cc17e499" (UID: "4474c9b2-6bac-4ef4-a223-0622cc17e499"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:13.812482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.812455    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv" pod "4474c9b2-6bac-4ef4-a223-0622cc17e499" (UID: "4474c9b2-6bac-4ef4-a223-0622cc17e499"). InnerVolumeSpecName "kube-api-access-fczlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:13.911401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.911369    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume\") pod \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") "
Nov 03 03:16:13.911637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.911446    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d\" (UniqueName: \"kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d\") pod \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\" (UID: \"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2\") "
Nov 03 03:16:13.911722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.911677    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fczlv\" (UniqueName: \"kubernetes.io/projected/4474c9b2-6bac-4ef4-a223-0622cc17e499-kube-api-access-fczlv\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:13.911722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.911689    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4474c9b2-6bac-4ef4-a223-0622cc17e499-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:13.911872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.911849    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume" pod "b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" (UID: "b5f598cb-0ac3-4468-af24-0e51f2a7a7e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:13.913751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:13.913729    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d" pod "b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" (UID: "b5f598cb-0ac3-4468-af24-0e51f2a7a7e2"). InnerVolumeSpecName "kube-api-access-7q28d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:14.012581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.012455    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:14.012581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.012487    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q28d\" (UniqueName: \"kubernetes.io/projected/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2-kube-api-access-7q28d\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:14.329150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.329069    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:14.467730 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.467694    5515 generic.go:356] "Generic (PLEG): container finished" podID="b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" containerID="616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96" exitCode=0
Nov 03 03:16:14.468217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.467741    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"
Nov 03 03:16:14.468217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.467752    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z" event={"ID":"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2","Type":"ContainerDied","Data":"616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96"}
Nov 03 03:16:14.468217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.467811    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z" event={"ID":"b5f598cb-0ac3-4468-af24-0e51f2a7a7e2","Type":"ContainerDied","Data":"68ff10de300388e988975f3819a04d46d8d03f81123c2e6e9d77df9e7ce4591f"}
Nov 03 03:16:14.468217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.467837    5515 scope.go:122] "RemoveContainer" containerID="616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96"
Nov 03 03:16:14.469561 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.469518    5515 generic.go:356] "Generic (PLEG): container finished" podID="4474c9b2-6bac-4ef4-a223-0622cc17e499" containerID="ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663" exitCode=0
Nov 03 03:16:14.469684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.469595    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg" event={"ID":"4474c9b2-6bac-4ef4-a223-0622cc17e499","Type":"ContainerDied","Data":"ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663"}
Nov 03 03:16:14.469684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.469606    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"
Nov 03 03:16:14.469684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.469633    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg" event={"ID":"4474c9b2-6bac-4ef4-a223-0622cc17e499","Type":"ContainerDied","Data":"767f95c53f532e483d519f812c59c738400d8e851170e0aac688645d6b71b7bb"}
Nov 03 03:16:14.478628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.478603    5515 scope.go:122] "RemoveContainer" containerID="616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96"
Nov 03 03:16:14.479036 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:14.479003    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96\": container with ID starting with 616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96 not found: ID does not exist" containerID="616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96"
Nov 03 03:16:14.479126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.479052    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96"} err="failed to get container status \"616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96\": rpc error: code = NotFound desc = could not find container \"616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96\": container with ID starting with 616a7ee18d614333c02da02c4a5fce18bc63fff90844f696b181d21ccd5c8c96 not found: ID does not exist"
Nov 03 03:16:14.479126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.479072    5515 scope.go:122] "RemoveContainer" containerID="ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663"
Nov 03 03:16:14.487855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.487828    5515 scope.go:122] "RemoveContainer" containerID="ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663"
Nov 03 03:16:14.488206 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:14.488181    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663\": container with ID starting with ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663 not found: ID does not exist" containerID="ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663"
Nov 03 03:16:14.488295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.488214    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663"} err="failed to get container status \"ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663\": rpc error: code = NotFound desc = could not find container \"ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663\": container with ID starting with ff796b8630fe5ea6cfad01304ad08f2252d76436b2345f7f1bd6f2edabbf1663 not found: ID does not exist"
Nov 03 03:16:14.539757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.539726    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:14.653688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.653653    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"]
Nov 03 03:16:14.655330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.655301    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4fb905b5-5fcf894b47-8xtmg"]
Nov 03 03:16:14.702887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.702853    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:14.769035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.768998    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"]
Nov 03 03:16:14.779714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.779684    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4fb905b5-77dc476c55-x2b4z"]
Nov 03 03:16:14.893285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.893250    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4474c9b2-6bac-4ef4-a223-0622cc17e499" path="/var/lib/kubelet/pods/4474c9b2-6bac-4ef4-a223-0622cc17e499/volumes"
Nov 03 03:16:14.893645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.893631    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" path="/var/lib/kubelet/pods/b5f598cb-0ac3-4468-af24-0e51f2a7a7e2/volumes"
Nov 03 03:16:14.974626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:14.974513    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:15.125698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:15.125662    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:15.271199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:15.271112    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:15.925007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:15.924975    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:16.138006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:16.137975    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"]
Nov 03 03:16:16.390598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:16.390536    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:16.541910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:16.541873    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:16.823461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:16.823335    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:18.302336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.302294    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:16:18.745842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.745819    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:16:18.860143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860113    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860177    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860198    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860221    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860246    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860318    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860368    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860484    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860524    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data\") pod \"199ab1e7-67f3-485c-b25c-fe8e088e754f\" (UID: \"199ab1e7-67f3-485c-b25c-fe8e088e754f\") "
Nov 03 03:16:18.860649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860537    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:18.860649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860585    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:18.860875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860818    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:18.860922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860903    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:18.860964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.860919    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:18.861085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.861069    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.861144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.861089    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199ab1e7-67f3-485c-b25c-fe8e088e754f-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.861144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.861100    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.861144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.861114    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.861144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.861125    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.862686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.862657    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:16:18.862890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.862867    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:18.863083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.863067    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "kube-api-access-rwqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:18.863145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.863068    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy" pod "199ab1e7-67f3-485c-b25c-fe8e088e754f" (UID: "199ab1e7-67f3-485c-b25c-fe8e088e754f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:18.962456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.962413    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwqdf\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-kube-api-access-rwqdf\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.962456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.962447    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.962456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.962460    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:18.962723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:18.962472    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199ab1e7-67f3-485c-b25c-fe8e088e754f-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:19.508755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.508723    5515 generic.go:356] "Generic (PLEG): container finished" podID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerID="caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20" exitCode=0
Nov 03 03:16:19.509273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.508770    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"
Nov 03 03:16:19.509273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.508813    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" event={"ID":"199ab1e7-67f3-485c-b25c-fe8e088e754f","Type":"ContainerDied","Data":"caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20"}
Nov 03 03:16:19.509273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.508874    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2" event={"ID":"199ab1e7-67f3-485c-b25c-fe8e088e754f","Type":"ContainerDied","Data":"c4639ec7beffd40c491070282682b0199c6ab4e321c0945a385a72e924552d47"}
Nov 03 03:16:19.509273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.508907    5515 scope.go:122] "RemoveContainer" containerID="caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20"
Nov 03 03:16:19.517719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.517695    5515 scope.go:122] "RemoveContainer" containerID="89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"
Nov 03 03:16:19.525905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.525882    5515 scope.go:122] "RemoveContainer" containerID="caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20"
Nov 03 03:16:19.526163 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:19.526143    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20\": container with ID starting with caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20 not found: ID does not exist" containerID="caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20"
Nov 03 03:16:19.526215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.526172    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20"} err="failed to get container status \"caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20\": rpc error: code = NotFound desc = could not find container \"caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20\": container with ID starting with caf68d745509b7ef2de756eb44ef44d176d40d7c1944e35f60181321063ace20 not found: ID does not exist"
Nov 03 03:16:19.526215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.526188    5515 scope.go:122] "RemoveContainer" containerID="89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"
Nov 03 03:16:19.526412 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:19.526392    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea\": container with ID starting with 89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea not found: ID does not exist" containerID="89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"
Nov 03 03:16:19.526452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.526427    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea"} err="failed to get container status \"89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea\": rpc error: code = NotFound desc = could not find container \"89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea\": container with ID starting with 89b7ca10169797149a1b741ee85cd251cb18c94c9bfe9f51ec8a84c5f64cd6ea not found: ID does not exist"
Nov 03 03:16:19.571579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.571506    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:19.676838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.676811    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"]
Nov 03 03:16:19.729607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.729571    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b4ff4779-openshift-default-655c4797d7-x4fk2"]
Nov 03 03:16:19.743132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.743101    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:19.909992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:19.909956    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:20.554261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:20.554226    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:20.735201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:20.735161    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:20.892305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:20.892272    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" path="/var/lib/kubelet/pods/199ab1e7-67f3-485c-b25c-fe8e088e754f/volumes"
Nov 03 03:16:20.903034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:20.903005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:20.984182 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:20.984146    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:16:21.139274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.139213    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng" podUID="44482ad5-99e6-4dce-9679-014b20e66e0a" containerName="payload-pre-processing" containerID="cri-o://86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b" gracePeriod=25
Nov 03 03:16:21.386963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.386940    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:16:21.492029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.491939    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq\" (UniqueName: \"kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq\") pod \"44482ad5-99e6-4dce-9679-014b20e66e0a\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") "
Nov 03 03:16:21.492029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.492000    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume\") pod \"44482ad5-99e6-4dce-9679-014b20e66e0a\" (UID: \"44482ad5-99e6-4dce-9679-014b20e66e0a\") "
Nov 03 03:16:21.492428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.492402    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume" pod "44482ad5-99e6-4dce-9679-014b20e66e0a" (UID: "44482ad5-99e6-4dce-9679-014b20e66e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:21.494230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.494206    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq" pod "44482ad5-99e6-4dce-9679-014b20e66e0a" (UID: "44482ad5-99e6-4dce-9679-014b20e66e0a"). InnerVolumeSpecName "kube-api-access-4brrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:21.520582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.520534    5515 generic.go:356] "Generic (PLEG): container finished" podID="44482ad5-99e6-4dce-9679-014b20e66e0a" containerID="86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b" exitCode=0
Nov 03 03:16:21.520734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.520590    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng" event={"ID":"44482ad5-99e6-4dce-9679-014b20e66e0a","Type":"ContainerDied","Data":"86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b"}
Nov 03 03:16:21.520734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.520612    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"
Nov 03 03:16:21.520734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.520644    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng" event={"ID":"44482ad5-99e6-4dce-9679-014b20e66e0a","Type":"ContainerDied","Data":"0e37dc4c4ec56a04efa92d2daa16fd634daa022324e14c9b768d29b8af2f91c7"}
Nov 03 03:16:21.520734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.520671    5515 scope.go:122] "RemoveContainer" containerID="86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b"
Nov 03 03:16:21.532251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.532226    5515 scope.go:122] "RemoveContainer" containerID="86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b"
Nov 03 03:16:21.533050 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:21.533021    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b\": container with ID starting with 86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b not found: ID does not exist" containerID="86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b"
Nov 03 03:16:21.533189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.533063    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b"} err="failed to get container status \"86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b\": rpc error: code = NotFound desc = could not find container \"86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b\": container with ID starting with 86cde8e2343cb5855e05ab4736f378c22ca0a42cb34d47be4f1a27ec7e99e93b not found: ID does not exist"
Nov 03 03:16:21.593274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.593238    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brrq\" (UniqueName: \"kubernetes.io/projected/44482ad5-99e6-4dce-9679-014b20e66e0a-kube-api-access-4brrq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:21.593274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.593267    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44482ad5-99e6-4dce-9679-014b20e66e0a-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:21.649210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.649176    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"]
Nov 03 03:16:21.653360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:21.653336    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56094c67-67c5466655-bt8ng"]
Nov 03 03:16:22.690241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:22.690213    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:22.832519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:22.832486    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:22.893481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:22.893449    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44482ad5-99e6-4dce-9679-014b20e66e0a" path="/var/lib/kubelet/pods/44482ad5-99e6-4dce-9679-014b20e66e0a/volumes"
Nov 03 03:16:24.663203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:24.663173    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:26.187956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.187913    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:26.364912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.364877    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:26.557596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.557489    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:26.757976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.757944    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"]
Nov 03 03:16:26.758407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758394    5515 cpu_manager.go:414] "containerMap: removing container" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758408    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758417    5515 cpu_manager.go:414] "containerMap: removing container" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758422    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758430    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" containerName="payload-pre-processing"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758436    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" containerName="payload-pre-processing"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758445    5515 cpu_manager.go:414] "containerMap: removing container" podUID="baec0252-144b-4dcb-b5d6-8fc3ddeca35d" containerName="payload-pre-processing"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758450    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baec0252-144b-4dcb-b5d6-8fc3ddeca35d" containerName="payload-pre-processing"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758457    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4474c9b2-6bac-4ef4-a223-0622cc17e499" containerName="payload-processing"
Nov 03 03:16:26.758461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758462    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4474c9b2-6bac-4ef4-a223-0622cc17e499" containerName="payload-processing"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758474    5515 cpu_manager.go:414] "containerMap: removing container" podUID="44482ad5-99e6-4dce-9679-014b20e66e0a" containerName="payload-pre-processing"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758479    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44482ad5-99e6-4dce-9679-014b20e66e0a" containerName="payload-pre-processing"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758534    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758541    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="199ab1e7-67f3-485c-b25c-fe8e088e754f" containerName="istio-proxy"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758562    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f598cb-0ac3-4468-af24-0e51f2a7a7e2" containerName="payload-pre-processing"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758572    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="baec0252-144b-4dcb-b5d6-8fc3ddeca35d" containerName="payload-pre-processing"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758581    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4474c9b2-6bac-4ef4-a223-0622cc17e499" containerName="payload-processing"
Nov 03 03:16:26.758782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.758590    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="44482ad5-99e6-4dce-9679-014b20e66e0a" containerName="payload-pre-processing"
Nov 03 03:16:26.762140 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.762123    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:26.820496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.820399    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"]
Nov 03 03:16:26.843395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.843361    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:26.843395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.843398    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpfqt\" (UniqueName: \"kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt\") pod \"payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:26.869304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.869276    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-de27f5\""
Nov 03 03:16:26.869442 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.869326    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-de27f5-dockercfg-k494x\""
Nov 03 03:16:26.944060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.944028    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:26.944060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.944062    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpfqt\" (UniqueName: \"kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt\") pod \"payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:26.944756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:26.944739    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:27.007631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.007602    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpfqt\" (UniqueName: \"kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt\") pod \"payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:27.077415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.077340    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:27.313056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.313022    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"]
Nov 03 03:16:27.449094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.449071    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"]
Nov 03 03:16:27.451401 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:27.451369    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266b0b26_9296_4a87_aa4e_a7ef5d3104ab.slice/crio-5ce52db27bc128b68cd80282d8780e60c4b0ee342c4ba565e896ab975744e1aa WatchSource:0}: Error finding container 5ce52db27bc128b68cd80282d8780e60c4b0ee342c4ba565e896ab975744e1aa: Status 404 returned error can't find the container with id 5ce52db27bc128b68cd80282d8780e60c4b0ee342c4ba565e896ab975744e1aa
Nov 03 03:16:27.552164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.552132    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" event={"ID":"266b0b26-9296-4a87-aa4e-a7ef5d3104ab","Type":"ContainerStarted","Data":"5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff"}
Nov 03 03:16:27.552356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.552195    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" event={"ID":"266b0b26-9296-4a87-aa4e-a7ef5d3104ab","Type":"ContainerStarted","Data":"5ce52db27bc128b68cd80282d8780e60c4b0ee342c4ba565e896ab975744e1aa"}
Nov 03 03:16:27.552356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.552223    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:27.673297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.673269    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:27.683205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.683161    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" podStartSLOduration=1.6831470259999999 podStartE2EDuration="1.683147026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:27.68238512 +0000 UTC m=+878.913277538" watchObservedRunningTime="2026-11-03 03:16:27.683147026 +0000 UTC m=+878.914039444"
Nov 03 03:16:27.817886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:27.817809    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:30.700067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.700038    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"]
Nov 03 03:16:30.700526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.700261    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" podUID="8351ee8e-50ff-48df-8f25-c9f709ea2dca" containerName="maas-api" containerID="cri-o://ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f" gracePeriod=30
Nov 03 03:16:30.798418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.798386    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:30.974006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.973982    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:16:30.983157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.983136    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs\") pod \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") "
Nov 03 03:16:30.983281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.983217    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca\") pod \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") "
Nov 03 03:16:30.983281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.983266    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b\" (UniqueName: \"kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b\") pod \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") "
Nov 03 03:16:30.983357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.983344    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\") pod \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\" (UID: \"8351ee8e-50ff-48df-8f25-c9f709ea2dca\") "
Nov 03 03:16:30.983666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.983634    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca" pod "8351ee8e-50ff-48df-8f25-c9f709ea2dca" (UID: "8351ee8e-50ff-48df-8f25-c9f709ea2dca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:30.985608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.985573    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b" pod "8351ee8e-50ff-48df-8f25-c9f709ea2dca" (UID: "8351ee8e-50ff-48df-8f25-c9f709ea2dca"). InnerVolumeSpecName "kube-api-access-vjk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:30.985712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.985635    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs" pod "8351ee8e-50ff-48df-8f25-c9f709ea2dca" (UID: "8351ee8e-50ff-48df-8f25-c9f709ea2dca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:16:30.985712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:30.985681    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls" pod "8351ee8e-50ff-48df-8f25-c9f709ea2dca" (UID: "8351ee8e-50ff-48df-8f25-c9f709ea2dca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:16:31.092539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.091207    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:31.092539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.091246    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351ee8e-50ff-48df-8f25-c9f709ea2dca-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:31.092539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.091261    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjk6b\" (UniqueName: \"kubernetes.io/projected/8351ee8e-50ff-48df-8f25-c9f709ea2dca-kube-api-access-vjk6b\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:31.092539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.091276    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351ee8e-50ff-48df-8f25-c9f709ea2dca-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:31.152752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.152718    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:31.328365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.328255    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:31.567950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.567915    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"]
Nov 03 03:16:31.568387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.568374    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8351ee8e-50ff-48df-8f25-c9f709ea2dca" containerName="maas-api"
Nov 03 03:16:31.568450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.568389    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8351ee8e-50ff-48df-8f25-c9f709ea2dca" containerName="maas-api"
Nov 03 03:16:31.568487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.568450    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8351ee8e-50ff-48df-8f25-c9f709ea2dca" containerName="maas-api"
Nov 03 03:16:31.571451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.571433    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.579755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.579726    5515 generic.go:356] "Generic (PLEG): container finished" podID="8351ee8e-50ff-48df-8f25-c9f709ea2dca" containerID="ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f" exitCode=0
Nov 03 03:16:31.579894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.579798    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"
Nov 03 03:16:31.579894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.579817    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" event={"ID":"8351ee8e-50ff-48df-8f25-c9f709ea2dca","Type":"ContainerDied","Data":"ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f"}
Nov 03 03:16:31.579894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.579864    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k" event={"ID":"8351ee8e-50ff-48df-8f25-c9f709ea2dca","Type":"ContainerDied","Data":"774a518f544ae3e90cefd5113d26fc4d710acb58426b189159ad26abbcae73e8"}
Nov 03 03:16:31.579894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.579885    5515 scope.go:122] "RemoveContainer" containerID="ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f"
Nov 03 03:16:31.589244 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.589225    5515 scope.go:122] "RemoveContainer" containerID="ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f"
Nov 03 03:16:31.589527 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:31.589506    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f\": container with ID starting with ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f not found: ID does not exist" containerID="ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f"
Nov 03 03:16:31.589667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.589539    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f"} err="failed to get container status \"ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f\": rpc error: code = NotFound desc = could not find container \"ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f\": container with ID starting with ed67d7d3379bcf83c161ddc4962165af743764bfcdd1f94dc1dc0ae63d5d519f not found: ID does not exist"
Nov 03 03:16:31.594199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.594178    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6lh\" (UniqueName: \"kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh\") pod \"payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.594297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.594215    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.628285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.628251    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-68b670-dockercfg-tnhxv\""
Nov 03 03:16:31.628473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.628303    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-68b670\""
Nov 03 03:16:31.630147 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.630119    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"]
Nov 03 03:16:31.695082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.695045    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6lh\" (UniqueName: \"kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh\") pod \"payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.695312 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.695091    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.695943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.695921    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.760886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.760859    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6lh\" (UniqueName: \"kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh\") pod \"payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:31.805448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.805410    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"]
Nov 03 03:16:31.811824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.811796    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56094c67-6ccf4b878d-7qg7k"]
Nov 03 03:16:31.885462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:31.885433    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:32.290151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.289827    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"]
Nov 03 03:16:32.292136 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:32.292102    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf262dc_2456_41aa_a3d1_5bcd7126583d.slice/crio-95f303a2ea687e513eb7aae88c733230787e4c10d1f95fd83356615a21d4526c WatchSource:0}: Error finding container 95f303a2ea687e513eb7aae88c733230787e4c10d1f95fd83356615a21d4526c: Status 404 returned error can't find the container with id 95f303a2ea687e513eb7aae88c733230787e4c10d1f95fd83356615a21d4526c
Nov 03 03:16:32.313524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.313483    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw" podUID="b1399fba-8170-4593-9916-432e5ca21223" containerName="payload-processing" containerID="cri-o://3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684" gracePeriod=25
Nov 03 03:16:32.368710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.368673    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:32.515729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.515668    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:32.566103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.566077    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:16:32.586957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.586913    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz" event={"ID":"8bf262dc-2456-41aa-a3d1-5bcd7126583d","Type":"ContainerStarted","Data":"71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2"}
Nov 03 03:16:32.587153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.586984    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz" event={"ID":"8bf262dc-2456-41aa-a3d1-5bcd7126583d","Type":"ContainerStarted","Data":"95f303a2ea687e513eb7aae88c733230787e4c10d1f95fd83356615a21d4526c"}
Nov 03 03:16:32.587153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.587016    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:32.588420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.588392    5515 generic.go:356] "Generic (PLEG): container finished" podID="b1399fba-8170-4593-9916-432e5ca21223" containerID="3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684" exitCode=0
Nov 03 03:16:32.588598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.588430    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"
Nov 03 03:16:32.588598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.588480    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw" event={"ID":"b1399fba-8170-4593-9916-432e5ca21223","Type":"ContainerDied","Data":"3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684"}
Nov 03 03:16:32.588598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.588534    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw" event={"ID":"b1399fba-8170-4593-9916-432e5ca21223","Type":"ContainerDied","Data":"a9faef9e38a3a13ba2ec3e092f7b148627fe91db7c865b5c49f56cff0d81628e"}
Nov 03 03:16:32.588598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.588592    5515 scope.go:122] "RemoveContainer" containerID="3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684"
Nov 03 03:16:32.600437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.600412    5515 scope.go:122] "RemoveContainer" containerID="3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684"
Nov 03 03:16:32.601041 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:32.601020    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684\": container with ID starting with 3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684 not found: ID does not exist" containerID="3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684"
Nov 03 03:16:32.601150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.601050    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684"} err="failed to get container status \"3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684\": rpc error: code = NotFound desc = could not find container \"3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684\": container with ID starting with 3f77776c5fa8b478c9f1adb56775424892f1cb19003504b35ecb5bd3d4cc7684 not found: ID does not exist"
Nov 03 03:16:32.602562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.602525    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6\" (UniqueName: \"kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6\") pod \"b1399fba-8170-4593-9916-432e5ca21223\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") "
Nov 03 03:16:32.604729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.604707    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6" pod "b1399fba-8170-4593-9916-432e5ca21223" (UID: "b1399fba-8170-4593-9916-432e5ca21223"). InnerVolumeSpecName "kube-api-access-m76x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:32.659650 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.659616    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:32.703272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.703233    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume\") pod \"b1399fba-8170-4593-9916-432e5ca21223\" (UID: \"b1399fba-8170-4593-9916-432e5ca21223\") "
Nov 03 03:16:32.703539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.703523    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m76x6\" (UniqueName: \"kubernetes.io/projected/b1399fba-8170-4593-9916-432e5ca21223-kube-api-access-m76x6\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:32.704062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.704036    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume" pod "b1399fba-8170-4593-9916-432e5ca21223" (UID: "b1399fba-8170-4593-9916-432e5ca21223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:32.804887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.804849    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1399fba-8170-4593-9916-432e5ca21223-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:32.811220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.811109    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz" podStartSLOduration=1.811095476 podStartE2EDuration="1.811095476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:32.810776218 +0000 UTC m=+884.041668668" watchObservedRunningTime="2026-11-03 03:16:32.811095476 +0000 UTC m=+884.041987891"
Nov 03 03:16:32.893284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:32.893252    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8351ee8e-50ff-48df-8f25-c9f709ea2dca" path="/var/lib/kubelet/pods/8351ee8e-50ff-48df-8f25-c9f709ea2dca/volumes"
Nov 03 03:16:33.014656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:33.013776    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"]
Nov 03 03:16:33.021427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:33.021395    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-69e611c0-7897b884f-q4htw"]
Nov 03 03:16:33.474575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:33.474508    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"]
Nov 03 03:16:33.474890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:33.474861    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy" containerID="cri-o://fc55d2db795452d655f98295ba910b9f25f054768e6818b767b6bf7768e1e66f" gracePeriod=30
Nov 03 03:16:33.497667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:33.497634    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:34.106467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.106430    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:16:34.337793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.337745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:34.893123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.893095    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1399fba-8170-4593-9916-432e5ca21223" path="/var/lib/kubelet/pods/b1399fba-8170-4593-9916-432e5ca21223/volumes"
Nov 03 03:16:34.893431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.893414    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"]
Nov 03 03:16:34.893917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.893900    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b1399fba-8170-4593-9916-432e5ca21223" containerName="payload-processing"
Nov 03 03:16:34.893917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.893915    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1399fba-8170-4593-9916-432e5ca21223" containerName="payload-processing"
Nov 03 03:16:34.894056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.893980    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1399fba-8170-4593-9916-432e5ca21223" containerName="payload-processing"
Nov 03 03:16:34.898362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.898345    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927317    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927368    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927453    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927480    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927538    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927598    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.927725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927681    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.928034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927787    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnz4\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.928034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.927853    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:34.961625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.961541    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"]
Nov 03 03:16:34.996385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:34.996347    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:35.010150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.010119    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0aa6b9-openshift-default-dockercfg-57xw2\""
Nov 03 03:16:35.029079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.028952    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnz4\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030605    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030666    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030715    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030757    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030781    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030878    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030906    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.030938    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.033380    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.033865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.033729    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.052999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.035480    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.052999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.036047    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.053799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.053768    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.058047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.058005    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.058679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.058614    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.091619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.091584    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnz4\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.093933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.093905    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token\") pod \"e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") " pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.161652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.161563    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:35.210969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.210932    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:35.649674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.649650    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"]
Nov 03 03:16:35.651775 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:35.651748    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8623e33_0051_4ac5_b370_53fc00aa6fe6.slice/crio-c1e919bca69e4335953d17e6f47f3c5484b1324428b1170c7469292291729adb WatchSource:0}: Error finding container c1e919bca69e4335953d17e6f47f3c5484b1324428b1170c7469292291729adb: Status 404 returned error can't find the container with id c1e919bca69e4335953d17e6f47f3c5484b1324428b1170c7469292291729adb
Nov 03 03:16:35.654186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.654152    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:35.654277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.654221    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:35.654277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:35.654255    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:36.614910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:36.614880    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" event={"ID":"f8623e33-0051-4ac5-b370-53fc00aa6fe6","Type":"ContainerStarted","Data":"968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"}
Nov 03 03:16:36.615411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:36.614933    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" event={"ID":"f8623e33-0051-4ac5-b370-53fc00aa6fe6","Type":"ContainerStarted","Data":"c1e919bca69e4335953d17e6f47f3c5484b1324428b1170c7469292291729adb"}
Nov 03 03:16:36.789702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:36.789651    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podStartSLOduration=2.78963653 podStartE2EDuration="2.78963653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:36.787998834 +0000 UTC m=+888.018891252" watchObservedRunningTime="2026-11-03 03:16:36.78963653 +0000 UTC m=+888.020528947"
Nov 03 03:16:37.211455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:37.211415    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:37.217164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:37.217134    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:37.619609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:37.619567    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:37.620769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:37.620750    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:16:38.080139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.080039    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:38.300908 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.300873    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:38.452459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.452426    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:38.562125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.562097    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 03:16:38.627728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.627697    5515 generic.go:356] "Generic (PLEG): container finished" podID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerID="fc55d2db795452d655f98295ba910b9f25f054768e6818b767b6bf7768e1e66f" exitCode=0
Nov 03 03:16:38.628111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.627772    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" event={"ID":"d9949878-41c3-44a4-9e5e-fc7430c43e4c","Type":"ContainerDied","Data":"fc55d2db795452d655f98295ba910b9f25f054768e6818b767b6bf7768e1e66f"}
Nov 03 03:16:38.628111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.627837    5515 scope.go:122] "RemoveContainer" containerID="d02dd1b3f20c9c6c18afdb8014dccb13f0728055345c9b6a05c2eefc4fa29a49"
Nov 03 03:16:38.746875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.746848    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:16:38.768901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.768867    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.768901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.768903    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.768925    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.768945    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769082    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769155    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769200    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769217    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:38.769302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769235    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:38.769302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769249    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769363    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket\") pod \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\" (UID: \"d9949878-41c3-44a4-9e5e-fc7430c43e4c\") "
Nov 03 03:16:38.769525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769366    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:38.769525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769396    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:38.769717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769661    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:38.769957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769937    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.770003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769962    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.770003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769977    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.770003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.769991    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.770084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.770005    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.771332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.771305    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:16:38.771497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.771477    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "kube-api-access-995vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:38.771990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.771964    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:16:38.771990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.771977    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token" pod "d9949878-41c3-44a4-9e5e-fc7430c43e4c" (UID: "d9949878-41c3-44a4-9e5e-fc7430c43e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:38.870702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.870669    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.870702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.870699    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.870702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.870708    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995vv\" (UniqueName: \"kubernetes.io/projected/d9949878-41c3-44a4-9e5e-fc7430c43e4c-kube-api-access-995vv\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:38.870994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:38.870717    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9949878-41c3-44a4-9e5e-fc7430c43e4c-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:39.634108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:39.634082    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"
Nov 03 03:16:39.634602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:39.634129    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g" event={"ID":"d9949878-41c3-44a4-9e5e-fc7430c43e4c","Type":"ContainerDied","Data":"330109b142d1e9c98c2569e5276a3a3b9392906116c1f9ed1b6554640f015de0"}
Nov 03 03:16:39.634602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:39.634193    5515 scope.go:122] "RemoveContainer" containerID="fc55d2db795452d655f98295ba910b9f25f054768e6818b767b6bf7768e1e66f"
Nov 03 03:16:39.853203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:39.853166    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"]
Nov 03 03:16:39.903821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:39.903745    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-56094c67-openshift-default-6d789587fc-xtw8g"]
Nov 03 03:16:40.434499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:40.434468    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:40.670012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:40.669979    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:40.832614 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:40.832510    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:40.893562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:40.893511    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" path="/var/lib/kubelet/pods/d9949878-41c3-44a4-9e5e-fc7430c43e4c/volumes"
Nov 03 03:16:43.535539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:43.535476    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:43.600801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:43.600775    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 03:16:43.745792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:43.745750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:43.980998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:43.980959    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:44.201870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.201836    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"]
Nov 03 03:16:44.202266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.202254    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy"
Nov 03 03:16:44.202325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.202268    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy"
Nov 03 03:16:44.202325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.202281    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy"
Nov 03 03:16:44.202325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.202286    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy"
Nov 03 03:16:44.202408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.202343    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy"
Nov 03 03:16:44.202408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.202351    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9949878-41c3-44a4-9e5e-fc7430c43e4c" containerName="istio-proxy"
Nov 03 03:16:44.206838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.206817    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.262289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.262204    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0aa6b9\""
Nov 03 03:16:44.264827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.264802    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"]
Nov 03 03:16:44.312467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.312427    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0aa6b9-dockercfg-lwxhv\""
Nov 03 03:16:44.330291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.330254    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrh7x\" (UniqueName: \"kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x\") pod \"payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.330494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.330423    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.331718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.331698    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"]
Nov 03 03:16:44.335334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.335318    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.393686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.393528    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"]
Nov 03 03:16:44.431808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.431768    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzch8\" (UniqueName: \"kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8\") pod \"payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.431964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.431902    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.432001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.431973    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrh7x\" (UniqueName: \"kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x\") pod \"payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.432032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.432000    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.432642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.432624    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.494894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.494859    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrh7x\" (UniqueName: \"kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x\") pod \"payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.520491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.520410    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:44.532874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.532844    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.533003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.532913    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzch8\" (UniqueName: \"kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8\") pod \"payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.533823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.533799    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.598272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.598238    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzch8\" (UniqueName: \"kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8\") pod \"payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.647802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.647771    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:44.893992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:44.893971    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"]
Nov 03 03:16:44.895591 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:44.895568    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dc0268_3d46_40d8_8e4b_d5299cd9d074.slice/crio-22075c80af8f87375ccda9e3397d5ce1cb138eb88a05ceca9e520baf47aa6e99 WatchSource:0}: Error finding container 22075c80af8f87375ccda9e3397d5ce1cb138eb88a05ceca9e520baf47aa6e99: Status 404 returned error can't find the container with id 22075c80af8f87375ccda9e3397d5ce1cb138eb88a05ceca9e520baf47aa6e99
Nov 03 03:16:45.020089 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.020068    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"]
Nov 03 03:16:45.021674 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:45.021649    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65523bf_94e1_48f5_8fe1_37baaf65636c.slice/crio-e4880df36093c5095f0ad22ee1dcfba1f83db175cf37e6cb483468d8d70fee97 WatchSource:0}: Error finding container e4880df36093c5095f0ad22ee1dcfba1f83db175cf37e6cb483468d8d70fee97: Status 404 returned error can't find the container with id e4880df36093c5095f0ad22ee1dcfba1f83db175cf37e6cb483468d8d70fee97
Nov 03 03:16:45.667078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.667041    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" event={"ID":"e65523bf-94e1-48f5-8fe1-37baaf65636c","Type":"ContainerStarted","Data":"dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b"}
Nov 03 03:16:45.667542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.667102    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" event={"ID":"e65523bf-94e1-48f5-8fe1-37baaf65636c","Type":"ContainerStarted","Data":"e4880df36093c5095f0ad22ee1dcfba1f83db175cf37e6cb483468d8d70fee97"}
Nov 03 03:16:45.667542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.667129    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:45.668734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.668709    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" event={"ID":"a0dc0268-3d46-40d8-8e4b-d5299cd9d074","Type":"ContainerStarted","Data":"c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e"}
Nov 03 03:16:45.668842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.668755    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" event={"ID":"a0dc0268-3d46-40d8-8e4b-d5299cd9d074","Type":"ContainerStarted","Data":"22075c80af8f87375ccda9e3397d5ce1cb138eb88a05ceca9e520baf47aa6e99"}
Nov 03 03:16:45.668842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.668803    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:45.785454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.785404    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" podStartSLOduration=1.785390861 podStartE2EDuration="1.785390861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:45.784517173 +0000 UTC m=+897.015409590" watchObservedRunningTime="2026-11-03 03:16:45.785390861 +0000 UTC m=+897.016283280"
Nov 03 03:16:45.917129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:45.913846    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" podStartSLOduration=1.9138249699999998 podStartE2EDuration="1.91382497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:45.912115029 +0000 UTC m=+897.143007450" watchObservedRunningTime="2026-11-03 03:16:45.91382497 +0000 UTC m=+897.144717389"
Nov 03 03:16:48.983953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:48.983924    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:16:48.984883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:48.984859    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:16:48.986374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:48.986358    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:16:48.986790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:48.986774    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:16:49.280454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.280369    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"]
Nov 03 03:16:49.280628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.280574    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v" podUID="ddd84fab-f048-41d1-b34d-11d27184f6c0" containerName="maas-api" containerID="cri-o://91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a" gracePeriod=30
Nov 03 03:16:49.429821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.429606    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:16:49.434790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.434757    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:49.565044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.564954    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:16:49.688684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.688643    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\") pod \"ddd84fab-f048-41d1-b34d-11d27184f6c0\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") "
Nov 03 03:16:49.688870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.688724    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j\" (UniqueName: \"kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j\") pod \"ddd84fab-f048-41d1-b34d-11d27184f6c0\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") "
Nov 03 03:16:49.688870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.688794    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca\") pod \"ddd84fab-f048-41d1-b34d-11d27184f6c0\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") "
Nov 03 03:16:49.688870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.688826    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs\") pod \"ddd84fab-f048-41d1-b34d-11d27184f6c0\" (UID: \"ddd84fab-f048-41d1-b34d-11d27184f6c0\") "
Nov 03 03:16:49.689195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.689172    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca" pod "ddd84fab-f048-41d1-b34d-11d27184f6c0" (UID: "ddd84fab-f048-41d1-b34d-11d27184f6c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:16:49.694199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694143    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls" pod "ddd84fab-f048-41d1-b34d-11d27184f6c0" (UID: "ddd84fab-f048-41d1-b34d-11d27184f6c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:16:49.694199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694164    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs" pod "ddd84fab-f048-41d1-b34d-11d27184f6c0" (UID: "ddd84fab-f048-41d1-b34d-11d27184f6c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:16:49.694199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694171    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j" pod "ddd84fab-f048-41d1-b34d-11d27184f6c0" (UID: "ddd84fab-f048-41d1-b34d-11d27184f6c0"). InnerVolumeSpecName "kube-api-access-7s89j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:49.694416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694289    5515 generic.go:356] "Generic (PLEG): container finished" podID="ddd84fab-f048-41d1-b34d-11d27184f6c0" containerID="91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a" exitCode=0
Nov 03 03:16:49.694416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694339    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"
Nov 03 03:16:49.694416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694371    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v" event={"ID":"ddd84fab-f048-41d1-b34d-11d27184f6c0","Type":"ContainerDied","Data":"91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a"}
Nov 03 03:16:49.694606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694463    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v" event={"ID":"ddd84fab-f048-41d1-b34d-11d27184f6c0","Type":"ContainerDied","Data":"41b9e9b2bd1c854c07072573b381d0ddf565ec406d307f08f728333232ad87b2"}
Nov 03 03:16:49.694606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.694508    5515 scope.go:122] "RemoveContainer" containerID="91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a"
Nov 03 03:16:49.715821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.715801    5515 scope.go:122] "RemoveContainer" containerID="91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a"
Nov 03 03:16:49.716158 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:49.716140    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a\": container with ID starting with 91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a not found: ID does not exist" containerID="91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a"
Nov 03 03:16:49.716253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.716167    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a"} err="failed to get container status \"91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a\": rpc error: code = NotFound desc = could not find container \"91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a\": container with ID starting with 91e5bbc9826a96ac0a9fb8bd6a39945cc4f84e1700d50a8679913e27e724ad7a not found: ID does not exist"
Nov 03 03:16:49.764518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.764472    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:49.789992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.789953    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:49.789992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.789987    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd84fab-f048-41d1-b34d-11d27184f6c0-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:49.789992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.789999    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s89j\" (UniqueName: \"kubernetes.io/projected/ddd84fab-f048-41d1-b34d-11d27184f6c0-kube-api-access-7s89j\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:49.790210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.790009    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd84fab-f048-41d1-b34d-11d27184f6c0-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:16:49.864115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.864078    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"]
Nov 03 03:16:49.865824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:49.865802    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-69e611c0-dd586fc8-8466v"]
Nov 03 03:16:50.069879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:50.069843    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:50.893260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:50.893226    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd84fab-f048-41d1-b34d-11d27184f6c0" path="/var/lib/kubelet/pods/ddd84fab-f048-41d1-b34d-11d27184f6c0/volumes"
Nov 03 03:16:50.985267 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:50.985234    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:16:51.039635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:51.039595    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:51.306413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:51.306333    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:51.458544 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:51.458508    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:51.739945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:51.739916    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:51.888940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:51.888905    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:52.558898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:52.558815    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:52.700784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:52.700750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:53.094446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.094407    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"]
Nov 03 03:16:53.094945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.094924    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ddd84fab-f048-41d1-b34d-11d27184f6c0" containerName="maas-api"
Nov 03 03:16:53.095036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.094947    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd84fab-f048-41d1-b34d-11d27184f6c0" containerName="maas-api"
Nov 03 03:16:53.095121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.095108    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd84fab-f048-41d1-b34d-11d27184f6c0" containerName="maas-api"
Nov 03 03:16:53.099375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.099349    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.155269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.155241    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5a6246-openshift-default-dockercfg-nmbkv\""
Nov 03 03:16:53.185401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.185374    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:53.207633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.207598    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"]
Nov 03 03:16:53.222709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.222674    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.232878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.232833    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.232884    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.232981    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.233054    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.233078    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.233106    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphsr\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.233130    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.233376 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.233317    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334241    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334304    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334341    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334360    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334531    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334643    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334675    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334711    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334719    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphsr\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334752    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334773    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.334985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.334951    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.335363 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.335338    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.335580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.335429    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.336772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.336745    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.337327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.337306    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.361136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.361102    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:53.400309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.400274    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.400472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.400315    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphsr\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.403949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.403921    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"]
Nov 03 03:16:53.408847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.408823    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.412565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.412527    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:53.465022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.463292    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b1ad4e-openshift-default-dockercfg-m64tn\""
Nov 03 03:16:53.476578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.472619    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"]
Nov 03 03:16:53.516799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.516707    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:53.541043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.540986    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv9g\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541072    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541133    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541221    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541290    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541321    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541380    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541420    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.541571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.541462    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.642994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.642892    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.642994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.642941    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.642994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.642988    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643013    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643032    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643088    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv9g\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643124    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643161    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643212    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643377    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643399    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643500    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643678    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.643900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.643790    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.645701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.645684    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.645819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.645801    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.706819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.706785    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.711203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.711178    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv9g\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.749604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.749541    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:53.762733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.762690    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:53.878414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.878388    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"]
Nov 03 03:16:53.879880 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:53.879852    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c56464_7dbd_411a_a63f_14fec9ea0c58.slice/crio-298014adb46c6ae94538b94f01ac61f04d529c9df1481ccfba8046a0856a9e82 WatchSource:0}: Error finding container 298014adb46c6ae94538b94f01ac61f04d529c9df1481ccfba8046a0856a9e82: Status 404 returned error can't find the container with id 298014adb46c6ae94538b94f01ac61f04d529c9df1481ccfba8046a0856a9e82
Nov 03 03:16:53.882232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.882196    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:53.882324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.882277    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:53.882401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:53.882322    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:54.148045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.147957    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"]
Nov 03 03:16:54.150943 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:54.150916    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb50507_ebc4_4f1f_a847_8d3f2ebc0a09.slice/crio-ebb1349758754bec2ee444c38b4e9cfc05cbcd700e30571834421d32d76bccfb WatchSource:0}: Error finding container ebb1349758754bec2ee444c38b4e9cfc05cbcd700e30571834421d32d76bccfb: Status 404 returned error can't find the container with id ebb1349758754bec2ee444c38b4e9cfc05cbcd700e30571834421d32d76bccfb
Nov 03 03:16:54.153446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.153407    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:54.153585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.153490    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:54.153585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.153532    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:16:54.722430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.722394    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" event={"ID":"f6c56464-7dbd-411a-a63f-14fec9ea0c58","Type":"ContainerStarted","Data":"647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"}
Nov 03 03:16:54.722916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.722452    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" event={"ID":"f6c56464-7dbd-411a-a63f-14fec9ea0c58","Type":"ContainerStarted","Data":"298014adb46c6ae94538b94f01ac61f04d529c9df1481ccfba8046a0856a9e82"}
Nov 03 03:16:54.730956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.730923    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" event={"ID":"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09","Type":"ContainerStarted","Data":"480a6f0238ae7c790427054b994fba11bc9ef2180b33aa4928ab4bc2461c78e5"}
Nov 03 03:16:54.731124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.730997    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" event={"ID":"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09","Type":"ContainerStarted","Data":"ebb1349758754bec2ee444c38b4e9cfc05cbcd700e30571834421d32d76bccfb"}
Nov 03 03:16:54.763018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.762981    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:54.847679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:54.847563    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podStartSLOduration=1.847531536 podStartE2EDuration="1.847531536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:54.845489036 +0000 UTC m=+906.076381469" watchObservedRunningTime="2026-11-03 03:16:54.847531536 +0000 UTC m=+906.078423999"
Nov 03 03:16:55.412907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:55.412869    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:55.417921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:55.417893    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:55.543245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:55.543190    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podStartSLOduration=2.543173481 podStartE2EDuration="2.543173481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:54.974107159 +0000 UTC m=+906.204999579" watchObservedRunningTime="2026-11-03 03:16:55.543173481 +0000 UTC m=+906.774065899"
Nov 03 03:16:55.736571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:55.736456    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:55.737428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:55.737409    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:16:55.768852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:55.768820    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:56.412926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.412891    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"]
Nov 03 03:16:56.416937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.416912    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.470964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.470930    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-0aa6b9-serving-cert\""
Nov 03 03:16:56.477017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.476988    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"]
Nov 03 03:16:56.480049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.480024    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvnb\" (UniqueName: \"kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.480220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.480057    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.480220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.480104    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.480336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.480239    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.581220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.581177    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvnb\" (UniqueName: \"kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.581220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.581221    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.581453 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:56.581322    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-serving-cert: secret "maas-api-e2e-worker-w0-0aa6b9-serving-cert" not found
Nov 03 03:16:56.581453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.581354    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.581453 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:16:56.581372    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls podName:0c8cd813-2c64-4ce0-876b-2aabb23af795 nodeName:}" failed. No retries permitted until 2026-11-03 03:16:57.081357235 +0000 UTC m=+908.312249632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls") pod "maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" (UID: "0c8cd813-2c64-4ce0-876b-2aabb23af795") : secret "maas-api-e2e-worker-w0-0aa6b9-serving-cert" not found
Nov 03 03:16:56.581453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.581395    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.582097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.582068    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.583925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.583901    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.611657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.611621    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:56.645239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.645200    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvnb\" (UniqueName: \"kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:56.678502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.678422    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 03:16:56.678502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.678484    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 03:16:56.740997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.740965    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:56.741750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.741731    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:16:56.801694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.801666    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:56.957210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:56.957121    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:57.087426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:57.087394    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:57.090173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:57.090136    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:57.330986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:57.330892    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:57.712068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:57.712041    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"]
Nov 03 03:16:57.713913 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:16:57.713889    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8cd813_2c64_4ce0_876b_2aabb23af795.slice/crio-8619b00c3fa0959583a87acbcb6ecffe14ee8d4ce9513090eeca56040ed63205 WatchSource:0}: Error finding container 8619b00c3fa0959583a87acbcb6ecffe14ee8d4ce9513090eeca56040ed63205: Status 404 returned error can't find the container with id 8619b00c3fa0959583a87acbcb6ecffe14ee8d4ce9513090eeca56040ed63205
Nov 03 03:16:57.746987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:57.746951    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" event={"ID":"0c8cd813-2c64-4ce0-876b-2aabb23af795","Type":"ContainerStarted","Data":"8619b00c3fa0959583a87acbcb6ecffe14ee8d4ce9513090eeca56040ed63205"}
Nov 03 03:16:58.755817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:58.755782    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" event={"ID":"0c8cd813-2c64-4ce0-876b-2aabb23af795","Type":"ContainerStarted","Data":"6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571"}
Nov 03 03:16:58.756313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:58.756039    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:16:58.875879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:58.875830    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" podStartSLOduration=2.013628149 podStartE2EDuration="2.875815403s" totalImagesPullingTime="862.187254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:58.875622582 +0000 UTC m=+910.106515000" watchObservedRunningTime="2026-11-03 03:16:58.875815403 +0000 UTC m=+910.106707821"
Nov 03 03:16:59.103498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:59.103466    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:59.329960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:59.329869    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:59.478709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:59.478668    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:59.704971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:59.704939    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:16:59.854386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:16:59.854355    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:00.100889 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:00.100809    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:00.251753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:00.251715    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:00.409931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:00.409896    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:00.562880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:00.562850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:02.383487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:02.383444    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:04.589385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:04.589348    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:04.767986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:04.767953    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:04.775753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:04.775732    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 03:17:05.999075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:05.999042    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:06.027152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:06.027120    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:17:07.120775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:07.120725    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:07.303095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:07.303060    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:07.740682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:07.740650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:11.235688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:11.235638    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:11.902107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:11.902073    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:12.060837 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:12.060810    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:12.210088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:12.210006    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:13.048529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:13.048493    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:13.700718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:13.700681    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:17:15.865959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:15.865924    5515 generic.go:356] "Generic (PLEG): container finished" podID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerID="968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da" exitCode=0
Nov 03 03:17:15.866507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:15.866003    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" event={"ID":"f8623e33-0051-4ac5-b370-53fc00aa6fe6","Type":"ContainerDied","Data":"968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"}
Nov 03 03:17:15.866507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:15.866486    5515 scope.go:122] "RemoveContainer" containerID="968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"
Nov 03 03:17:15.867834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:15.867804    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:15.867972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:15.867870    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:15.867972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:15.867903    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:16.896462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:16.896427    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" event={"ID":"f8623e33-0051-4ac5-b370-53fc00aa6fe6","Type":"ContainerStarted","Data":"0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273"}
Nov 03 03:17:17.211538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:17.211464    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:17:17.211685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:17.211662    5515 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 03:17:17.211746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:17.211727    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 03:17:17.987856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:17.987822    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:18.136901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:18.136857    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:18.212270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:18.212234    5515 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 03:17:18.212480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:18.212299    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 03:17:18.292484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:18.292394    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:19.211661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:19.211624    5515 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 03:17:19.212082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:19.211692    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 03:17:20.151317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:20.151278    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:20.212109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:20.212071    5515 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 03:17:20.212530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:20.212131    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 03:17:21.005695 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:17:21.005657    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:17:21.211857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:21.211819    5515 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 03:17:21.212043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:21.211873    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 03:17:22.215920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:22.215888    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:17:22.216439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:22.216311    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:17:22.216973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:22.216954    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 03:17:23.077747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:23.077715    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:23.239090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:23.239053    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:23.406847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:23.406813    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:28.247010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:28.246974    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:28.269623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:28.269587    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:17:31.786743 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:31.786704    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:33.230153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:33.230108    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:35.505448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:35.505413    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:36.262670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:36.262627    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:36.420463 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:36.420414    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:36.893807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:36.893771    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:17:38.021095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.021014    5515 generic.go:356] "Generic (PLEG): container finished" podID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerID="647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799" exitCode=0
Nov 03 03:17:38.021095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.021085    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" event={"ID":"f6c56464-7dbd-411a-a63f-14fec9ea0c58","Type":"ContainerDied","Data":"647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"}
Nov 03 03:17:38.021570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.021449    5515 scope.go:122] "RemoveContainer" containerID="647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"
Nov 03 03:17:38.022715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.022685    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:38.022835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.022769    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:38.022835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.022819    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:38.444610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:38.444580    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:39.028153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:39.028120    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" event={"ID":"f6c56464-7dbd-411a-a63f-14fec9ea0c58","Type":"ContainerStarted","Data":"04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453"}
Nov 03 03:17:39.412738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:39.412704    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:17:39.414142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:39.414110    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 03:17:39.414201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:39.414179    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 03:17:40.413177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:40.413132    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 03:17:40.413711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:40.413671    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 03:17:40.737027 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:40.736943    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:17:41.413137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:41.413096    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 03:17:41.413458 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:41.413426    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 03:17:42.413160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:42.413129    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 03:17:42.413471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:42.413439    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 03:17:43.413341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:43.413306    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 03 03:17:43.413536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:43.413373    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 03 03:17:43.438712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:43.438653    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:44.418082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:44.418040    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:17:44.419104 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:44.419080    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 03:17:45.069204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:45.069169    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:47.072031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:47.071951    5515 generic.go:356] "Generic (PLEG): container finished" podID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerID="480a6f0238ae7c790427054b994fba11bc9ef2180b33aa4928ab4bc2461c78e5" exitCode=0
Nov 03 03:17:47.072437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:47.072022    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" event={"ID":"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09","Type":"ContainerDied","Data":"480a6f0238ae7c790427054b994fba11bc9ef2180b33aa4928ab4bc2461c78e5"}
Nov 03 03:17:47.072437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:47.072418    5515 scope.go:122] "RemoveContainer" containerID="480a6f0238ae7c790427054b994fba11bc9ef2180b33aa4928ab4bc2461c78e5"
Nov 03 03:17:47.073725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:47.073698    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:47.073795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:47.073762    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:47.073830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:47.073795    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:17:48.090178 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:48.090138    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" event={"ID":"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09","Type":"ContainerStarted","Data":"e9f34293ffa79dca918608174baa4346d08aad23f217ee021c7fdc10a4c0887a"}
Nov 03 03:17:48.439142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:48.439112    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:48.787635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:48.783937    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:17:48.787635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:48.784092    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 03:17:48.787635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:48.784161    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 03:17:49.774671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:49.774635    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 03:17:49.775274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:49.775243    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 03:17:50.771576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:50.769906    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 03:17:50.771576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:50.769968    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 03:17:51.008739 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:17:51.008704    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:17:51.780137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:51.780093    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 03:17:51.781205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:51.781171    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 03:17:52.673602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:52.673189    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:52.766123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:52.766085    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 03:17:52.766415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:52.766386    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 03:17:52.943371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:52.943294    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:53.345807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:53.345706    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:53.767434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:53.767404    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:17:53.767957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:53.767827    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:17:53.768465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:53.768447    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:17:58.636332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:58.636299    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:58.889938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:58.889849    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:59.430011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:59.429984    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:59.591567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:59.591523    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:17:59.593723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:59.593698    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:17:59.747688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:17:59.747605    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:02.080591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.080543    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"]
Nov 03 03:18:02.084160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.084144    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.134061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.134027    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.134257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.134085    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.134257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.134195    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.134257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.134229    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.134401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.134287    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.134456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.134441    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gg8\" (UniqueName: \"kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.170751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.170714    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:02.186814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.186776    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"openshift-service-ca.crt\""
Nov 03 03:18:02.186969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.186832    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"kube-root-ca.crt\""
Nov 03 03:18:02.189058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.189040    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"simulator-w0-0aa6b9-kserve-self-signed-certs\""
Nov 03 03:18:02.189058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.189049    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"default-dockercfg-twz4w\""
Nov 03 03:18:02.190753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.190734    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"]
Nov 03 03:18:02.235695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.235661    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.235893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.235723    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.235893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.235762    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.235893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.235788    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.235893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.235837    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.236122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.235920    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gg8\" (UniqueName: \"kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.236185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.236157    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.236185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.236176    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.236318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.236300    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.238408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.238383    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.238606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.238591    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.298690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.298658    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gg8\" (UniqueName: \"kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8\") pod \"simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") " pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.396070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.396036    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:02.769375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:02.769348    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"]
Nov 03 03:18:02.771077 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:18:02.771049    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8627883_0bb5_4416_a42a_f0c5c9ef4252.slice/crio-7e2743fb6ed781f199a15221e7e4d9d0e0bb90560a7740a9df449ab62d624725 WatchSource:0}: Error finding container 7e2743fb6ed781f199a15221e7e4d9d0e0bb90560a7740a9df449ab62d624725: Status 404 returned error can't find the container with id 7e2743fb6ed781f199a15221e7e4d9d0e0bb90560a7740a9df449ab62d624725
Nov 03 03:18:03.168306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:03.168252    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" event={"ID":"d8627883-0bb5-4416-a42a-f0c5c9ef4252","Type":"ContainerStarted","Data":"26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6"}
Nov 03 03:18:03.168796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:03.168318    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" event={"ID":"d8627883-0bb5-4416-a42a-f0c5c9ef4252","Type":"ContainerStarted","Data":"7e2743fb6ed781f199a15221e7e4d9d0e0bb90560a7740a9df449ab62d624725"}
Nov 03 03:18:03.289148 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:03.289094    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podStartSLOduration=1.289079637 podStartE2EDuration="1.289079637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:03.287646846 +0000 UTC m=+974.518539265" watchObservedRunningTime="2026-11-03 03:18:03.289079637 +0000 UTC m=+974.519972055"
Nov 03 03:18:03.400105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:03.400051    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:03.930968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:03.930914    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:04.080226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:04.080193    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:04.630388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:04.630356    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:04.886039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:04.885911    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:05.041079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:05.041011    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:08.857905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:08.857879    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:12.396661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:12.396627    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:12.409374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:12.409346    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:13.230845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:13.230818    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:18:13.963200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:13.963157    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:14.529733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.529700    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"]
Nov 03 03:18:14.535146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.535114    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.593185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.593158    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"]
Nov 03 03:18:14.622195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.622162    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:14.642164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.642138    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"default-dockercfg-rjr5x\""
Nov 03 03:18:14.642164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.642146    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"simulator-w1-5a6246-kserve-self-signed-certs\""
Nov 03 03:18:14.642364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.642197    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"openshift-service-ca.crt\""
Nov 03 03:18:14.643899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.643878    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"kube-root-ca.crt\""
Nov 03 03:18:14.679976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.679927    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.679976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.679974    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.680268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.680008    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797zm\" (UniqueName: \"kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.680268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.680056    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.680268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.680110    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.680268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.680153    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.781686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.781605    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.781686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.781640    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.781686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.781664    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797zm\" (UniqueName: \"kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.781981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.781805    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.781981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.781872    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.781981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.781930    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.782131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.782073    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.782131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.782114    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.782274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.782255    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.784176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.784157    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.784563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.784527    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.844251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.844214    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797zm\" (UniqueName: \"kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm\") pod \"simulator-w1-5a6246-kserve-755d547c68-979q8\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") " pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:14.849153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:14.849128    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:15.230798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:15.230774    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"]
Nov 03 03:18:15.233292 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:18:15.233266    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb11e6b5c_300c_48e3_a427_446eddd74f52.slice/crio-a267fb6265b559240beba24afcffab9d4401ca352e6a845b7a9ccbc7c7cd485c WatchSource:0}: Error finding container a267fb6265b559240beba24afcffab9d4401ca352e6a845b7a9ccbc7c7cd485c: Status 404 returned error can't find the container with id a267fb6265b559240beba24afcffab9d4401ca352e6a845b7a9ccbc7c7cd485c
Nov 03 03:18:16.237964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:16.237933    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" event={"ID":"b11e6b5c-300c-48e3-a427-446eddd74f52","Type":"ContainerStarted","Data":"e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313"}
Nov 03 03:18:16.237964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:16.237971    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" event={"ID":"b11e6b5c-300c-48e3-a427-446eddd74f52","Type":"ContainerStarted","Data":"a267fb6265b559240beba24afcffab9d4401ca352e6a845b7a9ccbc7c7cd485c"}
Nov 03 03:18:16.359674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:16.359623    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podStartSLOduration=2.359607858 podStartE2EDuration="2.359607858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:16.358096989 +0000 UTC m=+987.588989409" watchObservedRunningTime="2026-11-03 03:18:16.359607858 +0000 UTC m=+987.590500276"
Nov 03 03:18:16.849419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:16.849386    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:18.940601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:18.940526    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:19.015896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:19.015864    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:18:20.984355 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:18:20.984313    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:18:23.940015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:23.939922    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:24.849873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:24.849838    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:24.862363 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:24.862340    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:25.294843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:25.294816    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:18:28.933679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:28.933630    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:33.941017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:33.940970    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:35.481155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:35.481116    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:38.836158 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:38.836090    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:41.320336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:41.320305    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:18:43.832841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:43.832736    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:44.685086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:44.685032    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:48.746482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:48.746447    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:50.983360 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:18:50.983325    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:18:53.298477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:53.298419    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:53.645342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:53.645300    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:58.303017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.302981    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"]
Nov 03 03:18:58.306672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.306648    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.392112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.392072    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:58.410978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.410952    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0aa6b9\"/\"premium-w0-0aa6b9-kserve-self-signed-certs\""
Nov 03 03:18:58.412785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.412767    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"]
Nov 03 03:18:58.457010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.456969    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.457480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.457450    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.457805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.457767    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.458065 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.458043    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s8w\" (UniqueName: \"kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.458331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.458176    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.458514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.458496    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.542794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.542767    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:18:58.559646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.559567    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s8w\" (UniqueName: \"kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.559646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.559604    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.559646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.559625    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.559894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.559677    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.559894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.559707    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.559894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.559861    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.560061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.560041    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.560120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.560086    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.560279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.560257    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.562152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.562129    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.562286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.562270    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.622384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.622347    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s8w\" (UniqueName: \"kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w\") pod \"premium-w0-0aa6b9-kserve-8d9c4748f-9lpms\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") " pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:58.917860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:58.917827    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:18:59.291070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:59.291047    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"]
Nov 03 03:18:59.293492 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:18:59.293467    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda89dd4ba_b869_4902_9cd3_968437773d1f.slice/crio-0d6e22763a7b00f9237d6962f21ddc9fd8770663f085551d6b6f5c6ae25f5085 WatchSource:0}: Error finding container 0d6e22763a7b00f9237d6962f21ddc9fd8770663f085551d6b6f5c6ae25f5085: Status 404 returned error can't find the container with id 0d6e22763a7b00f9237d6962f21ddc9fd8770663f085551d6b6f5c6ae25f5085
Nov 03 03:18:59.471809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:59.471768    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" event={"ID":"a89dd4ba-b869-4902-9cd3-968437773d1f","Type":"ContainerStarted","Data":"292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224"}
Nov 03 03:18:59.471809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:59.471808    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" event={"ID":"a89dd4ba-b869-4902-9cd3-968437773d1f","Type":"ContainerStarted","Data":"0d6e22763a7b00f9237d6962f21ddc9fd8770663f085551d6b6f5c6ae25f5085"}
Nov 03 03:18:59.593044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:59.592939    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podStartSLOduration=1.5929238589999999 podStartE2EDuration="1.592923859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:18:59.59174127 +0000 UTC m=+1030.822633689" watchObservedRunningTime="2026-11-03 03:18:59.592923859 +0000 UTC m=+1030.823816277"
Nov 03 03:18:59.918755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:18:59.918719    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:19:01.280483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.280445    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"]
Nov 03 03:19:01.284160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.284140    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.337733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.337704    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\""
Nov 03 03:19:01.385877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.385713    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.385877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.385780    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.385877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.385809    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.385877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.385880    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsnl\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.386165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.385909    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.386165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.385936    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.386165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.386157    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.386278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.386189    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.386278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.386223    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.390679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.390651    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"]
Nov 03 03:19:01.487076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487029    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487087    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487117    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487179    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsnl\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487216    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487247    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487321    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487380    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487422    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487465    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487638    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487696    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487724    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.487970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.487952    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.489892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.489876    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.490045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.490029    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.549673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.549599    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.549673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.549638    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsnl\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:01.597698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:01.597670    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:02.027441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.027420    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"]
Nov 03 03:19:02.029311 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:19:02.029282    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5995f7c2_777d_4518_ab7e_712a54ca57e4.slice/crio-4e0ef5218ff12be2f15302952dbaed3faba98e3b9710f9de1c6d2ebb665b8b85 WatchSource:0}: Error finding container 4e0ef5218ff12be2f15302952dbaed3faba98e3b9710f9de1c6d2ebb665b8b85: Status 404 returned error can't find the container with id 4e0ef5218ff12be2f15302952dbaed3faba98e3b9710f9de1c6d2ebb665b8b85
Nov 03 03:19:02.031713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.031680    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:19:02.031768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.031748    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:19:02.031807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.031777    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:19:02.488349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.488301    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" event={"ID":"5995f7c2-777d-4518-ab7e-712a54ca57e4","Type":"ContainerStarted","Data":"0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be"}
Nov 03 03:19:02.488773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.488368    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" event={"ID":"5995f7c2-777d-4518-ab7e-712a54ca57e4","Type":"ContainerStarted","Data":"4e0ef5218ff12be2f15302952dbaed3faba98e3b9710f9de1c6d2ebb665b8b85"}
Nov 03 03:19:02.598302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.598269    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:02.615431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:02.612474    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podStartSLOduration=1.612458555 podStartE2EDuration="1.612458555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:02.611427136 +0000 UTC m=+1033.842319555" watchObservedRunningTime="2026-11-03 03:19:02.612458555 +0000 UTC m=+1033.843350973"
Nov 03 03:19:03.435799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:03.435750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:03.517338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:03.516729    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:19:03.599451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:03.599420    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 03:19:03.599673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:03.599476    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 03:19:04.599507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:04.598989    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 03:19:04.599507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:04.599051    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 03:19:05.598073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:05.598030    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 03:19:05.598281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:05.598092    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 03:19:06.598329 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:06.598298    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 03:19:06.598751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:06.598357    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 03:19:07.598025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:07.597992    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 03:19:07.598207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:07.598051    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 03:19:08.337054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:08.337019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:08.918877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:08.918839    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:19:08.932269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:08.932244    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:19:09.415264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:09.415228    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:09.415869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:09.415673    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:09.417223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:09.417204    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:19:09.564371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:09.564339    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:19:13.247986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:13.247944    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:18.140538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:18.140497    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:20.984430 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:19:20.984378    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:19:23.029611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:23.029572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:24.346403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:24.346372    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:26.563444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:26.563411    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:19:27.964172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:27.964136    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:32.845742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:32.845691    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:37.738788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:37.738750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:41.699490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:41.699443    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:42.824837 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:42.824790    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:47.702207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:47.702154    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:48.279704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.279673    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"]
Nov 03 03:19:48.283659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.283636    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.341399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.341372    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-68b670-openshift-default-dockercfg-tnxvt\""
Nov 03 03:19:48.382079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382038    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382091    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpf6\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382198    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382226    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382252    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382330    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382414    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382457    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.382655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.382507    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.390320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.390297    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"]
Nov 03 03:19:48.483800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.483764    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.483800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.483807    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpf6\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.483843    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.483862    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.483879    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.483900    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484019    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484065    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484111    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484418    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484534    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484664    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484667    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.484753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.484696    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.486614 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.486597    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.486672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.486617    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.546594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.546504    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.546719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.546593    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpf6\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.597231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.597205    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:48.870262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.870232    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"]
Nov 03 03:19:48.870683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:48.870424    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy" containerID="cri-o://7278c0bf23244f406a6119f026237f46f1d6201c971a1fd22f09776bd1139277" gracePeriod=30
Nov 03 03:19:49.039172 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:19:49.039148    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64709aa4_be02_4056_a5a7_756f4c6bad1b.slice/crio-351a0c579e588ad0d998273c9fea8c83dfc10e7acd81e0716fe5ff43f2207c10 WatchSource:0}: Error finding container 351a0c579e588ad0d998273c9fea8c83dfc10e7acd81e0716fe5ff43f2207c10: Status 404 returned error can't find the container with id 351a0c579e588ad0d998273c9fea8c83dfc10e7acd81e0716fe5ff43f2207c10
Nov 03 03:19:49.041346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.041312    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:19:49.041442 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.041397    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:19:49.041504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.041441    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:19:49.048544 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.048523    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"]
Nov 03 03:19:49.790022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.789983    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" event={"ID":"64709aa4-be02-4056-a5a7-756f4c6bad1b","Type":"ContainerStarted","Data":"fe69845a8ac5286473ce1b281b955ffef0117d44a1817919e5d81a52c2c4e22e"}
Nov 03 03:19:49.790203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.790045    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" event={"ID":"64709aa4-be02-4056-a5a7-756f4c6bad1b","Type":"ContainerStarted","Data":"351a0c579e588ad0d998273c9fea8c83dfc10e7acd81e0716fe5ff43f2207c10"}
Nov 03 03:19:49.912765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:49.912711    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podStartSLOduration=1.912696494 podStartE2EDuration="1.912696494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:19:49.911740158 +0000 UTC m=+1081.142632572" watchObservedRunningTime="2026-11-03 03:19:49.912696494 +0000 UTC m=+1081.143588911"
Nov 03 03:19:50.597962 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:50.597921    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:50.599535 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:50.599513    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 03:19:50.599626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:50.599591    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 03:19:51.009157 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:19:51.009120    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:19:51.319218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:51.319135    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:19:51.387928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:51.387870    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:51.536535 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:51.536484    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:51.598941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:51.598858    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 03:19:51.598941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:51.598925    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 03:19:52.600634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:52.599876    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 03:19:52.600634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:52.599936    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 03:19:52.765574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:52.757120    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:53.598087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:53.598053    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 03:19:53.598284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:53.598098    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 03:19:53.648529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:53.648491    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:19:53.916671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:53.916642    5515 generic.go:356] "Generic (PLEG): container finished" podID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerID="7278c0bf23244f406a6119f026237f46f1d6201c971a1fd22f09776bd1139277" exitCode=0
Nov 03 03:19:53.916828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:53.916715    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" event={"ID":"79a23be2-6f70-4b11-88fe-443fcb40f732","Type":"ContainerDied","Data":"7278c0bf23244f406a6119f026237f46f1d6201c971a1fd22f09776bd1139277"}
Nov 03 03:19:54.553737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.553714    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:19:54.599240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.599199    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 03:19:54.599438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.599259    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 03:19:54.732855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.732774    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.732855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.732813    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.732855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.732845    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.732955    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733134    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733156    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733169    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733222    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733235    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733253    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733286    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733311    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn\") pod \"79a23be2-6f70-4b11-88fe-443fcb40f732\" (UID: \"79a23be2-6f70-4b11-88fe-443fcb40f732\") "
Nov 03 03:19:54.733422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733341    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:19:54.733922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733585    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:19:54.733922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733810    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.733922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733830    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.733922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733844    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.733922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733860    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79a23be2-6f70-4b11-88fe-443fcb40f732-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.733922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.733873    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.736053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.736025    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:19:54.736215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.736194    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:19:54.736714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.736690    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:19:54.736813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.736724    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn" pod "79a23be2-6f70-4b11-88fe-443fcb40f732" (UID: "79a23be2-6f70-4b11-88fe-443fcb40f732"). InnerVolumeSpecName "kube-api-access-b9hdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:19:54.836027 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.835992    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.836027 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.836027    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.836272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.836044    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hdn\" (UniqueName: \"kubernetes.io/projected/79a23be2-6f70-4b11-88fe-443fcb40f732-kube-api-access-b9hdn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.836272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.836058    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79a23be2-6f70-4b11-88fe-443fcb40f732-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:19:54.923051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.923026    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"
Nov 03 03:19:54.923201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.923060    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8" event={"ID":"79a23be2-6f70-4b11-88fe-443fcb40f732","Type":"ContainerDied","Data":"0f665192118ecedcd25b83383ac1fed901af1b9bc54bf3319bc2ab42bd40a5ae"}
Nov 03 03:19:54.923201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:54.923122    5515 scope.go:122] "RemoveContainer" containerID="7278c0bf23244f406a6119f026237f46f1d6201c971a1fd22f09776bd1139277"
Nov 03 03:19:55.099101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:55.099013    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"]
Nov 03 03:19:55.103258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:55.103233    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-42kt8"]
Nov 03 03:19:55.597927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:55.597890    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 03 03:19:55.598110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:55.597954    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 03 03:19:56.602241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:56.602214    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:56.602776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:56.602590    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:56.603176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:56.603158    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:19:56.893336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:56.893306    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" path="/var/lib/kubelet/pods/79a23be2-6f70-4b11-88fe-443fcb40f732/volumes"
Nov 03 03:19:57.739290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:19:57.739256    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:02.838572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:02.838518    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:07.707277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:07.707246    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:12.735861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:12.735828    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:17.638095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:17.638050    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:20.984009 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:20:20.983967    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:20:22.561856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:22.561819    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:24.926948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:24.926920    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:25.076910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:25.076872    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:25.227627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:25.227537    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:27.554343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:27.554312    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:29.238989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:29.238955    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:30.409857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.409817    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"]
Nov 03 03:20:30.412412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.412384    5515 cpu_manager.go:414] "containerMap: removing container" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy"
Nov 03 03:20:30.412581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.412431    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy"
Nov 03 03:20:30.412748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.412727    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a23be2-6f70-4b11-88fe-443fcb40f732" containerName="istio-proxy"
Nov 03 03:20:30.417767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.417736    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.453423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.453387    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.453625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.453444    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.453680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.453653    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.453792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.453771    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpd5\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.470624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.470591    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"]
Nov 03 03:20:30.519369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.519334    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 03:20:30.519369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.519347    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-576a0931ff85\""
Nov 03 03:20:30.519690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.519389    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 03:20:30.519690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.519433    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6czs8\""
Nov 03 03:20:30.522515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.522489    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6867f294d875\""
Nov 03 03:20:30.522515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.522499    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 03:20:30.522695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.522527    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 03:20:30.554368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.554340    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.554526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.554416    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpd5\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.554526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.554487    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.554685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.554562    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.555287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.555264    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.555719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.555691    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.557359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.557337    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.617398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.617363    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpd5\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5\") pod \"payload-processing-external-model-56844d9c9c-sfxqn\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") " pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.734615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.734503    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:30.975428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:30.975399    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:20:31.112414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:31.112388    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"]
Nov 03 03:20:31.115154 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:20:31.115127    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd03c700_b8a6_43ad_9fb5_73afc930100c.slice/crio-5b2ed11eb0836d449aba95b79e8ca4f54eec6bd0627f2ce53a3f40e3de9e8c7b WatchSource:0}: Error finding container 5b2ed11eb0836d449aba95b79e8ca4f54eec6bd0627f2ce53a3f40e3de9e8c7b: Status 404 returned error can't find the container with id 5b2ed11eb0836d449aba95b79e8ca4f54eec6bd0627f2ce53a3f40e3de9e8c7b
Nov 03 03:20:31.142707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:31.142662    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn" event={"ID":"dd03c700-b8a6-43ad-9fb5-73afc930100c","Type":"ContainerStarted","Data":"5b2ed11eb0836d449aba95b79e8ca4f54eec6bd0627f2ce53a3f40e3de9e8c7b"}
Nov 03 03:20:32.150408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:32.150369    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn" event={"ID":"dd03c700-b8a6-43ad-9fb5-73afc930100c","Type":"ContainerStarted","Data":"4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249"}
Nov 03 03:20:32.150845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:32.150580    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:32.280045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:32.279990    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn" podStartSLOduration=2.2799720199999998 podStartE2EDuration="2.27997202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:32.277860272 +0000 UTC m=+1123.508752694" watchObservedRunningTime="2026-11-03 03:20:32.27997202 +0000 UTC m=+1123.510864438"
Nov 03 03:20:32.569694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:32.569615    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:37.523588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:37.523541    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:42.452905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:42.452872    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:43.159172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:43.159143    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:20:47.345053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:47.345015    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:49.378638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.378597    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"]
Nov 03 03:20:49.385060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.385030    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.427504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.427476    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"]
Nov 03 03:20:49.431251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.431232    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.459316 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.459278    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"]
Nov 03 03:20:49.485222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485191    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485249    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485289    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485391    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485417    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.485477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485453    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.485618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485503    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485531    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485584    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rs5\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.485618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485606    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.485807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485625    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.485807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485652    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485829    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfh6f\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.485946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485880    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.486002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485937    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.486002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.485985    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.486106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.486023    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.486106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.486082    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.538945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.538911    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"]
Nov 03 03:20:49.586737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586704    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.586737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586743    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.586957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586810    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.586957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586847    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.586957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586892    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.586957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586922    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.586957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586949    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.587176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.586988    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rs5\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587026    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587060    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587089    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.587176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587131    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfh6f\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.587176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587158    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587181    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.587397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587186    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587231    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587270    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.587397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587337    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.587397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587374    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.587967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.587945    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.588301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.588275    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.588430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.588366    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.588799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.588540    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.589483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.589454    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.589767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.589739    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.590263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.589955    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.590263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.590217    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.591145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.591118    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.591856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.591814    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.592230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.592210    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.594071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.594046    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.594239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.594219    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.670441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.667494    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfh6f\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.670441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.667977    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-dpj77\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.670441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.670332    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.673000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.672953    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rs5\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5\") pod \"maas-default-gateway-openshift-default-76f4876667-f6b4k\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:49.706726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.706690    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:49.744387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:49.744255    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:50.074914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.074813    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:50.172231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.171048    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"]
Nov 03 03:20:50.178122 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:20:50.178082    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6785670a_a73e_44dd_87b0_5abc41c720b1.slice/crio-80fd95c44411dfbdb3da8de9cfb818ce4a7fade8b404c5ff5213683e6299f333 WatchSource:0}: Error finding container 80fd95c44411dfbdb3da8de9cfb818ce4a7fade8b404c5ff5213683e6299f333: Status 404 returned error can't find the container with id 80fd95c44411dfbdb3da8de9cfb818ce4a7fade8b404c5ff5213683e6299f333
Nov 03 03:20:50.182954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.180978    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:20:50.182954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.181069    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:20:50.182954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.181116    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:20:50.252706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.252678    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"]
Nov 03 03:20:50.259885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.259694    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" event={"ID":"6785670a-a73e-44dd-87b0-5abc41c720b1","Type":"ContainerStarted","Data":"80fd95c44411dfbdb3da8de9cfb818ce4a7fade8b404c5ff5213683e6299f333"}
Nov 03 03:20:50.269800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.269750    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:20:50.269921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.269848    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:20:50.269921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:50.269897    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:20:51.034111 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:20:51.034069    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:20:51.303744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.303648    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" event={"ID":"6785670a-a73e-44dd-87b0-5abc41c720b1","Type":"ContainerStarted","Data":"348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1"}
Nov 03 03:20:51.324395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.324352    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" event={"ID":"6d39bd5c-bd30-4447-9c98-0c96719fcdb2","Type":"ContainerStarted","Data":"4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13"}
Nov 03 03:20:51.324739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.324717    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" event={"ID":"6d39bd5c-bd30-4447-9c98-0c96719fcdb2","Type":"ContainerStarted","Data":"893340b1ff5f9d90ef5da11e2124a5d1178c3209a3fff8efcf3c7c74bcfdedbd"}
Nov 03 03:20:51.556133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.553678    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podStartSLOduration=2.553652179 podStartE2EDuration="2.553652179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:51.491012864 +0000 UTC m=+1142.721905286" watchObservedRunningTime="2026-11-03 03:20:51.553652179 +0000 UTC m=+1142.784544656"
Nov 03 03:20:51.642064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.641986    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podStartSLOduration=2.641965714 podStartE2EDuration="2.641965714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:51.620057347 +0000 UTC m=+1142.850949777" watchObservedRunningTime="2026-11-03 03:20:51.641965714 +0000 UTC m=+1142.872858131"
Nov 03 03:20:51.708574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.707721    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:51.712570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.709880    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:51.712570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.709942    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:51.744871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.744827    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:51.746692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.746648    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:51.746847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:51.746705    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:52.296323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:52.296289    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:52.717575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:52.717029    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:52.717575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:52.717091    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:52.744733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:52.744612    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:52.744733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:52.744671    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:53.715934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:53.715467    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:53.715934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:53.715523    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:53.745371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:53.745333    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:53.745517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:53.745384    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:54.023138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:54.023041    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:54.712426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:54.712391    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:54.712775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:54.712749    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:54.745386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:54.745337    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:54.745819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:54.745639    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:55.708571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:55.707996    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:55.708571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:55.708055    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:55.747571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:55.747210    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:55.747571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:55.747286    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:56.707853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:56.707819    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:56.708037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:56.707880    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:56.744875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:56.744840    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:56.745048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:56.744908    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:57.091447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.091364    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-655f966974-z6wfs"]
Nov 03 03:20:57.095332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.095312    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.204251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.204214    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-655f966974-z6wfs"]
Nov 03 03:20:57.243806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.243761    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw5r\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.244013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.243887    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.244013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.243986    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.244151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.244074    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.327242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.327207    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:20:57.345671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.345596    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw5r\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.345671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.345655    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.345904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.345683    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.345904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.345734    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.346431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.346408    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.346535 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.346499    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.348592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.348570    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.408379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.408342    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw5r\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r\") pod \"payload-processing-external-model-655f966974-z6wfs\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") " pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.408974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.408954    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:57.707151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.707120    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-dpj77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 03:20:57.707355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.707182    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 03:20:57.745119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.745088    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-f6b4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 03:20:57.745278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.745138    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 03:20:57.792939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:57.792918    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-655f966974-z6wfs"]
Nov 03 03:20:57.795906 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:20:57.795879    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900d082a_976f_41e8_b3b7_954f6c95081f.slice/crio-52764f144f197bf0dcbc6531cf43347ab3e504383b85b3d7de94bb4ab3bac672 WatchSource:0}: Error finding container 52764f144f197bf0dcbc6531cf43347ab3e504383b85b3d7de94bb4ab3bac672: Status 404 returned error can't find the container with id 52764f144f197bf0dcbc6531cf43347ab3e504383b85b3d7de94bb4ab3bac672
Nov 03 03:20:58.400073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:58.400035    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" event={"ID":"900d082a-976f-41e8-b3b7-954f6c95081f","Type":"ContainerStarted","Data":"bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891"}
Nov 03 03:20:58.400631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:58.400107    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" event={"ID":"900d082a-976f-41e8-b3b7-954f6c95081f","Type":"ContainerStarted","Data":"52764f144f197bf0dcbc6531cf43347ab3e504383b85b3d7de94bb4ab3bac672"}
Nov 03 03:20:58.400631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:58.400225    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:20:58.524336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:58.524266    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" podStartSLOduration=1.524244615 podStartE2EDuration="1.524244615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:58.520808457 +0000 UTC m=+1149.751700878" watchObservedRunningTime="2026-11-03 03:20:58.524244615 +0000 UTC m=+1149.755137033"
Nov 03 03:20:59.216472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:59.216427    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:59.216882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:59.216859    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:59.217635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:59.217613    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:20:59.220912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:59.220765    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:59.221339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:59.221317    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:20:59.222632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:20:59.222612    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:21:02.256609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:02.256574    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:05.701355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:05.701323    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:21:07.166197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:07.166159    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:09.437917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:09.437882    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:21:12.238637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:12.238605    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:17.024702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.024665    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-655f966974-z6wfs"]
Nov 03 03:21:17.026083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.026055    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:21:17.075237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.075208    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"]
Nov 03 03:21:17.079317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.079300    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.115365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.115336    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874jp\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.115490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.115394    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.115531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.115494    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.115598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.115542    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.124292 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.124251    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:17.125395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.125368    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"]
Nov 03 03:21:17.217296 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.217255    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874jp\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.217501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.217324    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.217501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.217355    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.217501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.217395    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.218093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.218069    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.218290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.218271    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.220027 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.220006    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.279450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.279384    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874jp\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp\") pod \"payload-processing-external-model-6575c6b868-x6hlx\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") " pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.392861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.392821    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:17.766888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:17.766860    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"]
Nov 03 03:21:17.768215 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:21:17.768190    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2100b7_1b1f_4595_9916_4a87aedd2b1b.slice/crio-1bc00c3ce2fbb7dbf5afc5f54b48c7b07c397e937d44b9417ca9faff6b20ddbc WatchSource:0}: Error finding container 1bc00c3ce2fbb7dbf5afc5f54b48c7b07c397e937d44b9417ca9faff6b20ddbc: Status 404 returned error can't find the container with id 1bc00c3ce2fbb7dbf5afc5f54b48c7b07c397e937d44b9417ca9faff6b20ddbc
Nov 03 03:21:18.535091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:18.535051    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" event={"ID":"fe2100b7-1b1f-4595-9916-4a87aedd2b1b","Type":"ContainerStarted","Data":"533cc4f5b0d8f09838df5451299258b135629e92cc08dd071106f461d5500bfe"}
Nov 03 03:21:18.535487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:18.535135    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" event={"ID":"fe2100b7-1b1f-4595-9916-4a87aedd2b1b","Type":"ContainerStarted","Data":"1bc00c3ce2fbb7dbf5afc5f54b48c7b07c397e937d44b9417ca9faff6b20ddbc"}
Nov 03 03:21:18.535487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:18.535160    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:18.653460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:18.653405    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" podStartSLOduration=1.6533928850000001 podStartE2EDuration="1.653392885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:18.652933144 +0000 UTC m=+1169.883825566" watchObservedRunningTime="2026-11-03 03:21:18.653392885 +0000 UTC m=+1169.884285304"
Nov 03 03:21:20.984168 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:20.984128    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:21:21.959945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:21.959906    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:22.026685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.026630    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" containerName="payload-processing" containerID="cri-o://bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891" gracePeriod=25
Nov 03 03:21:22.336433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.336404    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:21:22.375022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.374983    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume\") pod \"900d082a-976f-41e8-b3b7-954f6c95081f\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") "
Nov 03 03:21:22.375214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.375054    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials\") pod \"900d082a-976f-41e8-b3b7-954f6c95081f\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") "
Nov 03 03:21:22.375214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.375143    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume\") pod \"900d082a-976f-41e8-b3b7-954f6c95081f\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") "
Nov 03 03:21:22.375359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.375311    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r\") pod \"900d082a-976f-41e8-b3b7-954f6c95081f\" (UID: \"900d082a-976f-41e8-b3b7-954f6c95081f\") "
Nov 03 03:21:22.375500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.375474    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume" pod "900d082a-976f-41e8-b3b7-954f6c95081f" (UID: "900d082a-976f-41e8-b3b7-954f6c95081f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:21:22.375622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.375592    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume" pod "900d082a-976f-41e8-b3b7-954f6c95081f" (UID: "900d082a-976f-41e8-b3b7-954f6c95081f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:21:22.376025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.375994    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:21:22.376150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.376030    5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/900d082a-976f-41e8-b3b7-954f6c95081f-routing-overlay-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:21:22.377763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.377739    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r" pod "900d082a-976f-41e8-b3b7-954f6c95081f" (UID: "900d082a-976f-41e8-b3b7-954f6c95081f"). InnerVolumeSpecName "kube-api-access-6sw5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:21:22.377909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.377891    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials" pod "900d082a-976f-41e8-b3b7-954f6c95081f" (UID: "900d082a-976f-41e8-b3b7-954f6c95081f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:21:22.502844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.482476    5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-provider-credentials\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:21:22.502844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.482514    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sw5r\" (UniqueName: \"kubernetes.io/projected/900d082a-976f-41e8-b3b7-954f6c95081f-kube-api-access-6sw5r\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:21:22.559862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.559827    5515 generic.go:356] "Generic (PLEG): container finished" podID="900d082a-976f-41e8-b3b7-954f6c95081f" containerID="bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891" exitCode=0
Nov 03 03:21:22.560059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.559884    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs"
Nov 03 03:21:22.560059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.559915    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" event={"ID":"900d082a-976f-41e8-b3b7-954f6c95081f","Type":"ContainerDied","Data":"bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891"}
Nov 03 03:21:22.560059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.559969    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-655f966974-z6wfs" event={"ID":"900d082a-976f-41e8-b3b7-954f6c95081f","Type":"ContainerDied","Data":"52764f144f197bf0dcbc6531cf43347ab3e504383b85b3d7de94bb4ab3bac672"}
Nov 03 03:21:22.560059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.559991    5515 scope.go:122] "RemoveContainer" containerID="bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891"
Nov 03 03:21:22.630541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.611491    5515 scope.go:122] "RemoveContainer" containerID="bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891"
Nov 03 03:21:22.630541 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:22.618939    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891\": container with ID starting with bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891 not found: ID does not exist" containerID="bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891"
Nov 03 03:21:22.630541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.618985    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891"} err="failed to get container status \"bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891\": rpc error: code = NotFound desc = could not find container \"bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891\": container with ID starting with bca3220bb615db29ad0f6c629c7410ef30255e59be48bf8ee76e95df7fc2c891 not found: ID does not exist"
Nov 03 03:21:22.786053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.785967    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-655f966974-z6wfs"]
Nov 03 03:21:22.815657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.815624    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-655f966974-z6wfs"]
Nov 03 03:21:22.922132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:22.922097    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" path="/var/lib/kubelet/pods/900d082a-976f-41e8-b3b7-954f6c95081f/volumes"
Nov 03 03:21:25.673113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.673081    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"]
Nov 03 03:21:25.673600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.673571    5515 cpu_manager.go:414] "containerMap: removing container" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" containerName="payload-processing"
Nov 03 03:21:25.673600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.673585    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" containerName="payload-processing"
Nov 03 03:21:25.673697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.673689    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="900d082a-976f-41e8-b3b7-954f6c95081f" containerName="payload-processing"
Nov 03 03:21:25.678300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.678284    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.772853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.772818    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"]
Nov 03 03:21:25.862974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.862938    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.862979    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6gw\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863070    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863121    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863184    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863218    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863253    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863351    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.863395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.863385    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.964982    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965026    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6gw\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965061    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965101    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965130    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965163    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965205    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965260    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965300    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965642    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965748    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965771    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965752    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.965936 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.965829    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.967440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.967419    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:25.967764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:25.967745    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:26.027855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.027817    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:26.028009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.027930    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6gw\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:26.290515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.290437    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:26.719635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.719612    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"]
Nov 03 03:21:26.720466 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:21:26.720443    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7887d4e_c7a4_47c1_8b7c_41b52865d5c7.slice/crio-1046ed08ef04496f4cfd31eb33f56b2de5da6b815cf04031e1f0446321f88592 WatchSource:0}: Error finding container 1046ed08ef04496f4cfd31eb33f56b2de5da6b815cf04031e1f0446321f88592: Status 404 returned error can't find the container with id 1046ed08ef04496f4cfd31eb33f56b2de5da6b815cf04031e1f0446321f88592
Nov 03 03:21:26.722644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.722613    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:21:26.722702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.722677    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:21:26.722735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.722706    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:21:26.937270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:26.937238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:27.634960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:27.634921    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" event={"ID":"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7","Type":"ContainerStarted","Data":"e1c32524d1c0c940b9deca8cde4c9f132fe4303b0e089ef0579c3485ce9b318c"}
Nov 03 03:21:27.635183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:27.634998    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" event={"ID":"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7","Type":"ContainerStarted","Data":"1046ed08ef04496f4cfd31eb33f56b2de5da6b815cf04031e1f0446321f88592"}
Nov 03 03:21:27.858985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:27.858927    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podStartSLOduration=2.858912063 podStartE2EDuration="2.858912063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:27.807989349 +0000 UTC m=+1179.038881766" watchObservedRunningTime="2026-11-03 03:21:27.858912063 +0000 UTC m=+1179.089804480"
Nov 03 03:21:28.290873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:28.290836    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:28.292513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:28.292485    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 03:21:28.292674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:28.292532    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 03:21:29.290957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:29.290925    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 03:21:29.291366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:29.290998    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 03:21:29.544403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:29.544315    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:21:30.290915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:30.290872    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 03:21:30.291119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:30.290931    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 03:21:31.291332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:31.291291    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 03:21:31.291750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:31.291356    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 03:21:31.988258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:31.988220    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:32.290817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:32.290720    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 03:21:32.290817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:32.290780    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 03:21:33.291756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:33.291721    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 03:21:33.292227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:33.291787    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 03:21:33.478074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:33.478040    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:33.644863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:33.644819    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:34.295330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:34.295301    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:34.295791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:34.295770    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:34.296501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:34.296475    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:21:36.835657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:36.835626    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:37.171690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:37.171646    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:39.543191 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:39.543157    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:21:41.756590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:41.756535    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:44.995195 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:44.995155    5515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-576a0931ff85: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:44.995195 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:44.995195    5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:44.995874 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:44.995301    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials podName:dd03c700-b8a6-43ad-9fb5-73afc930100c nodeName:}" failed. No retries permitted until 2026-11-03 03:21:45.495280701 +0000 UTC m=+1196.726173110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials") pod "payload-processing-external-model-56844d9c9c-sfxqn" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c") : secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:45.501505 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:45.501470    5515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-576a0931ff85: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:45.501505 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:45.501501    5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:45.501772 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:45.501595    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials podName:dd03c700-b8a6-43ad-9fb5-73afc930100c nodeName:}" failed. No retries permitted until 2026-11-03 03:21:46.501578992 +0000 UTC m=+1197.732471401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials") pod "payload-processing-external-model-56844d9c9c-sfxqn" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c") : secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:46.288523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.288481    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:21:46.326270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.326237    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"]
Nov 03 03:21:46.330299 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.330282    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.386736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.386684    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"distinct-w1-5a6246-kserve-self-signed-certs\""
Nov 03 03:21:46.389278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.389246    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"]
Nov 03 03:21:46.413386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.413344    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.413598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.413421    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.413668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.413622    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.413668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.413661    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.413817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.413755    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntpmz\" (UniqueName: \"kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.414016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.413977    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.427580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.427524    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:46.517092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.517051    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.517351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.517333    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.518181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.518153    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntpmz\" (UniqueName: \"kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.518661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.518612    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.518900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.518866    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.519089 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.519073    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.519262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.519241    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.520132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.520105    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.521186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.520354    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.521186 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:46.520450    5515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-576a0931ff85: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:46.521186 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:46.520469    5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:46.521186 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:46.520579    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials podName:dd03c700-b8a6-43ad-9fb5-73afc930100c nodeName:}" failed. No retries permitted until 2026-11-03 03:21:48.520523533 +0000 UTC m=+1199.751415935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials") pod "payload-processing-external-model-56844d9c9c-sfxqn" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c") : secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:46.524481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.524441    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.524693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.524673    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.582160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.582086    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntpmz\" (UniqueName: \"kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz\") pod \"distinct-w1-5a6246-kserve-569b85464-bl7fl\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:46.638812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.638771    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:46.643651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:46.643630    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:47.022119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:47.022092    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"]
Nov 03 03:21:47.023582 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:21:47.023534    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod131361f6_42d9_40a0_adb7_927c17ecf9b0.slice/crio-3f016dca5e384a1e130a2ba92ebac4a3e22225d873e5def991748e2fe502ab88 WatchSource:0}: Error finding container 3f016dca5e384a1e130a2ba92ebac4a3e22225d873e5def991748e2fe502ab88: Status 404 returned error can't find the container with id 3f016dca5e384a1e130a2ba92ebac4a3e22225d873e5def991748e2fe502ab88
Nov 03 03:21:47.760696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:47.760662    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" event={"ID":"131361f6-42d9-40a0-adb7-927c17ecf9b0","Type":"ContainerStarted","Data":"b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6"}
Nov 03 03:21:47.760696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:47.760703    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" event={"ID":"131361f6-42d9-40a0-adb7-927c17ecf9b0","Type":"ContainerStarted","Data":"3f016dca5e384a1e130a2ba92ebac4a3e22225d873e5def991748e2fe502ab88"}
Nov 03 03:21:47.879911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:47.879863    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podStartSLOduration=1.879849715 podStartE2EDuration="1.879849715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:21:47.879499362 +0000 UTC m=+1199.110391781" watchObservedRunningTime="2026-11-03 03:21:47.879849715 +0000 UTC m=+1199.110742133"
Nov 03 03:21:48.549702 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:48.549667    5515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-576a0931ff85: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:48.549702 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:48.549704    5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:48.549925 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:48.549764    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials podName:dd03c700-b8a6-43ad-9fb5-73afc930100c nodeName:}" failed. No retries permitted until 2026-11-03 03:21:52.549748305 +0000 UTC m=+1203.780640720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials") pod "payload-processing-external-model-56844d9c9c-sfxqn" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c") : secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:48.644669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:48.644636    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:49.026579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:49.026531    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:21:49.028461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:49.028443    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:21:49.030073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:49.030055    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:21:49.032300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:49.032282    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:21:49.542796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:49.542755    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:21:49.889961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:49.889930    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:50.037223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:50.037185    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:50.182207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:50.182119    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:50.985046 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:50.985015    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:21:51.562275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:51.562231    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:52.599449 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:52.599418    5515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-576a0931ff85: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:52.599449 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:52.599447    5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn: secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:52.599985 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:21:52.599499    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials podName:dd03c700-b8a6-43ad-9fb5-73afc930100c nodeName:}" failed. No retries permitted until 2026-11-03 03:22:00.599485438 +0000 UTC m=+1211.830377835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials") pod "payload-processing-external-model-56844d9c9c-sfxqn" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c") : secret "agc-cred-576a0931ff85" not found
Nov 03 03:21:54.000970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:54.000938    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:54.332802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:54.332716    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"]
Nov 03 03:21:54.333982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:54.333956    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:21:54.356237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:54.356213    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:56.430309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:56.430275    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:21:56.644090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:56.644052    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:56.656475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:56.656450    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:56.822756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:56.822675    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:21:59.333120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.333025    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing" containerID="cri-o://533cc4f5b0d8f09838df5451299258b135629e92cc08dd071106f461d5500bfe" gracePeriod=25
Nov 03 03:21:59.852279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.852252    5515 generic.go:356] "Generic (PLEG): container finished" podID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerID="533cc4f5b0d8f09838df5451299258b135629e92cc08dd071106f461d5500bfe" exitCode=0
Nov 03 03:21:59.852396 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.852382    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" event={"ID":"fe2100b7-1b1f-4595-9916-4a87aedd2b1b","Type":"ContainerDied","Data":"533cc4f5b0d8f09838df5451299258b135629e92cc08dd071106f461d5500bfe"}
Nov 03 03:21:59.852443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.852424    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx" event={"ID":"fe2100b7-1b1f-4595-9916-4a87aedd2b1b","Type":"ContainerDied","Data":"1bc00c3ce2fbb7dbf5afc5f54b48c7b07c397e937d44b9417ca9faff6b20ddbc"}
Nov 03 03:21:59.852443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.852441    5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc00c3ce2fbb7dbf5afc5f54b48c7b07c397e937d44b9417ca9faff6b20ddbc"
Nov 03 03:21:59.909074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.908779    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:21:59.956504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.956471    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp\") pod \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") "
Nov 03 03:21:59.956717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.956513    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume\") pod \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") "
Nov 03 03:21:59.956717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.956624    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials\") pod \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") "
Nov 03 03:21:59.956717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.956658    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume\") pod \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\" (UID: \"fe2100b7-1b1f-4595-9916-4a87aedd2b1b\") "
Nov 03 03:21:59.956990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.956957    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume" pod "fe2100b7-1b1f-4595-9916-4a87aedd2b1b" (UID: "fe2100b7-1b1f-4595-9916-4a87aedd2b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:21:59.957122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.957070    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume" pod "fe2100b7-1b1f-4595-9916-4a87aedd2b1b" (UID: "fe2100b7-1b1f-4595-9916-4a87aedd2b1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:21:59.957176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.957164    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:21:59.957216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.957178    5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-routing-overlay-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:21:59.958825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.958801    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp" pod "fe2100b7-1b1f-4595-9916-4a87aedd2b1b" (UID: "fe2100b7-1b1f-4595-9916-4a87aedd2b1b"). InnerVolumeSpecName "kube-api-access-874jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:21:59.958941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:21:59.958833    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials" pod "fe2100b7-1b1f-4595-9916-4a87aedd2b1b" (UID: "fe2100b7-1b1f-4595-9916-4a87aedd2b1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:22:00.058207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:00.058172    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874jp\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-kube-api-access-874jp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:22:00.058207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:00.058206    5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe2100b7-1b1f-4595-9916-4a87aedd2b1b-provider-credentials\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:22:00.685222 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:22:00.675476    5515 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-576a0931ff85: secret "agc-cred-576a0931ff85" not found
Nov 03 03:22:00.685222 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:22:00.675516    5515 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn: secret "agc-cred-576a0931ff85" not found
Nov 03 03:22:00.685222 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:22:00.675619    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials podName:dd03c700-b8a6-43ad-9fb5-73afc930100c nodeName:}" failed. No retries permitted until 2026-11-03 03:22:16.675597634 +0000 UTC m=+1227.906490049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials") pod "payload-processing-external-model-56844d9c9c-sfxqn" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c") : secret "agc-cred-576a0931ff85" not found
Nov 03 03:22:00.857993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:00.857961    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"
Nov 03 03:22:01.081124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:01.080973    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"]
Nov 03 03:22:01.129283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:01.129250    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6575c6b868-x6hlx"]
Nov 03 03:22:01.558955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:01.558919    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:02.894043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:02.894009    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" path="/var/lib/kubelet/pods/fe2100b7-1b1f-4595-9916-4a87aedd2b1b/volumes"
Nov 03 03:22:03.018972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.018937    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"]
Nov 03 03:22:03.019437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.019421    5515 cpu_manager.go:414] "containerMap: removing container" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing"
Nov 03 03:22:03.019437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.019438    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing"
Nov 03 03:22:03.019585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.019524    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2100b7-1b1f-4595-9916-4a87aedd2b1b" containerName="payload-processing"
Nov 03 03:22:03.022950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.022930    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.104523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.104492    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:03.109833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.109804    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.109974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.109838    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.109974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.109879    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.110068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.110002    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.110124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.110068    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srk8\" (UniqueName: \"kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.110183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.110154    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.131239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.131215    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"openshift-service-ca.crt\""
Nov 03 03:22:03.131239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.131232    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"kube-root-ca.crt\""
Nov 03 03:22:03.131239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.131243    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"default-dockercfg-qtmq4\""
Nov 03 03:22:03.131457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.131326    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"e2e-conflict-b-e5f308a1-kserve-self-signed-certs\""
Nov 03 03:22:03.133354 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.133329    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"]
Nov 03 03:22:03.211627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.211514    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.212177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212143    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.212458 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212064    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.214317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212591    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.214317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212602    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.214317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212698    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.214317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212780    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srk8\" (UniqueName: \"kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.214317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.212831    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.215050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.214977    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.216126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.216100    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.216657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.216629    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.277978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.277941    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srk8\" (UniqueName: \"kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8\") pod \"e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") " pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.335468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.335443    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:03.754938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.754912    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"]
Nov 03 03:22:03.756412 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:22:03.756385    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54e952bf_fd36_44c3_b97c_a1d40130b4f8.slice/crio-5b911a5e7c7b6b1c53421c756dc83fbe64cbf3e632cfadb21dc31b5a6c3e0231 WatchSource:0}: Error finding container 5b911a5e7c7b6b1c53421c756dc83fbe64cbf3e632cfadb21dc31b5a6c3e0231: Status 404 returned error can't find the container with id 5b911a5e7c7b6b1c53421c756dc83fbe64cbf3e632cfadb21dc31b5a6c3e0231
Nov 03 03:22:03.877093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.877060    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv" event={"ID":"54e952bf-fd36-44c3-b97c-a1d40130b4f8","Type":"ContainerStarted","Data":"de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87"}
Nov 03 03:22:03.877237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.877105    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv" event={"ID":"54e952bf-fd36-44c3-b97c-a1d40130b4f8","Type":"ContainerStarted","Data":"5b911a5e7c7b6b1c53421c756dc83fbe64cbf3e632cfadb21dc31b5a6c3e0231"}
Nov 03 03:22:03.997759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:03.997708    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv" podStartSLOduration=1.997693634 podStartE2EDuration="1.997693634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:03.995522717 +0000 UTC m=+1215.226415148" watchObservedRunningTime="2026-11-03 03:22:03.997693634 +0000 UTC m=+1215.228586051"
Nov 03 03:22:04.336172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:04.336136    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:06.424874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:06.424847    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:06.551610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:06.551123    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"]
Nov 03 03:22:11.322630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.322594    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:11.552357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.552289    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn" podUID="dd03c700-b8a6-43ad-9fb5-73afc930100c" containerName="payload-processing" containerID="cri-o://4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249" gracePeriod=25
Nov 03 03:22:11.825409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.825383    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:22:11.921991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.921953    5515 generic.go:356] "Generic (PLEG): container finished" podID="dd03c700-b8a6-43ad-9fb5-73afc930100c" containerID="4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249" exitCode=0
Nov 03 03:22:11.922179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.922006    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"
Nov 03 03:22:11.922179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.922036    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn" event={"ID":"dd03c700-b8a6-43ad-9fb5-73afc930100c","Type":"ContainerDied","Data":"4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249"}
Nov 03 03:22:11.922179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.922089    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn" event={"ID":"dd03c700-b8a6-43ad-9fb5-73afc930100c","Type":"ContainerDied","Data":"5b2ed11eb0836d449aba95b79e8ca4f54eec6bd0627f2ce53a3f40e3de9e8c7b"}
Nov 03 03:22:11.922179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.922111    5515 scope.go:122] "RemoveContainer" containerID="4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249"
Nov 03 03:22:11.931060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.931044    5515 scope.go:122] "RemoveContainer" containerID="4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249"
Nov 03 03:22:11.931328 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:22:11.931311    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249\": container with ID starting with 4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249 not found: ID does not exist" containerID="4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249"
Nov 03 03:22:11.931394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.931337    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249"} err="failed to get container status \"4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249\": rpc error: code = NotFound desc = could not find container \"4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249\": container with ID starting with 4599eb5e647fa1a43bde86bf2cc26d8549bfd14bd7215ebffc99ccab12c9a249 not found: ID does not exist"
Nov 03 03:22:11.932682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.932664    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume\") pod \"dd03c700-b8a6-43ad-9fb5-73afc930100c\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") "
Nov 03 03:22:11.932773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.932761    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5\") pod \"dd03c700-b8a6-43ad-9fb5-73afc930100c\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") "
Nov 03 03:22:11.932902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.932887    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials\") pod \"dd03c700-b8a6-43ad-9fb5-73afc930100c\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") "
Nov 03 03:22:11.932964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.932935    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume\") pod \"dd03c700-b8a6-43ad-9fb5-73afc930100c\" (UID: \"dd03c700-b8a6-43ad-9fb5-73afc930100c\") "
Nov 03 03:22:11.933026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.933006    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume" pod "dd03c700-b8a6-43ad-9fb5-73afc930100c" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:22:11.933345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.933282    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume" pod "dd03c700-b8a6-43ad-9fb5-73afc930100c" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:22:11.933345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.933324    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:22:11.934843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.934821    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5" pod "dd03c700-b8a6-43ad-9fb5-73afc930100c" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c"). InnerVolumeSpecName "kube-api-access-svpd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:22:11.935033 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:11.935014    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials" pod "dd03c700-b8a6-43ad-9fb5-73afc930100c" (UID: "dd03c700-b8a6-43ad-9fb5-73afc930100c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:22:12.034898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:12.034863    5515 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-provider-credentials\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:22:12.034898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:12.034892    5515 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd03c700-b8a6-43ad-9fb5-73afc930100c-routing-overlay-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:22:12.034898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:12.034903    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svpd5\" (UniqueName: \"kubernetes.io/projected/dd03c700-b8a6-43ad-9fb5-73afc930100c-kube-api-access-svpd5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:22:12.347570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:12.347509    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"]
Nov 03 03:22:12.398417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:12.398386    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56844d9c9c-sfxqn"]
Nov 03 03:22:12.894185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:12.894150    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd03c700-b8a6-43ad-9fb5-73afc930100c" path="/var/lib/kubelet/pods/dd03c700-b8a6-43ad-9fb5-73afc930100c/volumes"
Nov 03 03:22:13.335703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:13.335621    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:13.348663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:13.348637    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:13.946135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:13.946107    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:22:15.654568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:15.654512    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:22:16.259835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:16.259797    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:20.983506 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:22:20.983464    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:22:21.526737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:21.526695    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:26.427814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:26.427765    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:31.246177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:31.246141    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:36.126933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:36.126893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:41.021177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:41.021145    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:45.512030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.511985    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"]
Nov 03 03:22:45.512809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.512788    5515 cpu_manager.go:414] "containerMap: removing container" podUID="dd03c700-b8a6-43ad-9fb5-73afc930100c" containerName="payload-processing"
Nov 03 03:22:45.512918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.512810    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd03c700-b8a6-43ad-9fb5-73afc930100c" containerName="payload-processing"
Nov 03 03:22:45.512973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.512951    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd03c700-b8a6-43ad-9fb5-73afc930100c" containerName="payload-processing"
Nov 03 03:22:45.516119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.516097    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.561959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.561924    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.562143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.561969    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.562143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.562058    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.562143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.562127    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97f7\" (UniqueName: \"kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.562332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.562217    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.562398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.562328    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.602865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.602821    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:45.622225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.622196    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"openshift-service-ca.crt\""
Nov 03 03:22:45.622225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.622213    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"kube-root-ca.crt\""
Nov 03 03:22:45.622418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.622287    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"distinct-2-w3-68b670-kserve-self-signed-certs\""
Nov 03 03:22:45.622418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.622285    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-68b670\"/\"default-dockercfg-x2m5k\""
Nov 03 03:22:45.624392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.624373    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"]
Nov 03 03:22:45.663699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.663646    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.688941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.688912    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.689113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.689099    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.689243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.689230    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.689387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.689374    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.690586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.690542    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97f7\" (UniqueName: \"kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.698586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.691539    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.699733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.699007    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.699902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.699269    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.700007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.690204    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.708726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.708699    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.753302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.753272    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97f7\" (UniqueName: \"kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7\") pod \"distinct-2-w3-68b670-kserve-b9d959b84-5mwzv\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") " pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.827788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.827698    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:45.925912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:45.925879    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:46.201968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:46.201942    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"]
Nov 03 03:22:46.203377 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:22:46.203350    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b96b271_e20b_4178_a7c5_8290d0aed2b5.slice/crio-613f0b1bd3b19aad277deeda497cf8173cba68e2ce1285025d6b21f4024316da WatchSource:0}: Error finding container 613f0b1bd3b19aad277deeda497cf8173cba68e2ce1285025d6b21f4024316da: Status 404 returned error can't find the container with id 613f0b1bd3b19aad277deeda497cf8173cba68e2ce1285025d6b21f4024316da
Nov 03 03:22:47.176649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:47.176612    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" event={"ID":"2b96b271-e20b-4178-a7c5-8290d0aed2b5","Type":"ContainerStarted","Data":"2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2"}
Nov 03 03:22:47.178891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:47.176655    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" event={"ID":"2b96b271-e20b-4178-a7c5-8290d0aed2b5","Type":"ContainerStarted","Data":"613f0b1bd3b19aad277deeda497cf8173cba68e2ce1285025d6b21f4024316da"}
Nov 03 03:22:47.297062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:47.297009    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podStartSLOduration=2.296996345 podStartE2EDuration="2.296996345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:22:47.294858597 +0000 UTC m=+1258.525751016" watchObservedRunningTime="2026-11-03 03:22:47.296996345 +0000 UTC m=+1258.527888760"
Nov 03 03:22:47.828697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:47.828666    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:50.830556 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:50.830513    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:50.983134 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:22:50.983101    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:22:54.793189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:54.793116    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:54.865921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:54.865890    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:22:55.744504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:55.744465    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:22:55.830033 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:55.829280    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:55.857229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:55.857203    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:56.239975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:56.239946    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:22:57.969969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:22:57.969933    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:00.635381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:00.635349    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:05.528153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:05.528113    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:10.071446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:10.071404    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:10.428052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:10.428006    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:15.311886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:15.311848    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:20.229685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:20.229651    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:20.984813 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:23:20.984777    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:23:25.161764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:25.161726    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:26.417295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:26.417262    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:23:27.036740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.036712    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"]
Nov 03 03:23:27.041000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.040984    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.111591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.111540    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.111757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.111689    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.111810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.111761    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.111810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.111794    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.111887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.111864    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.111984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.111969    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mps5\" (UniqueName: \"kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.126601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.126574    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:27.146462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.146438    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5a6246\"/\"distinct-2-w1-5a6246-kserve-self-signed-certs\""
Nov 03 03:23:27.148592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.148572    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"]
Nov 03 03:23:27.222348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.222313    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.222637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.222614    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.224418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.223129    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.224536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.223780    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.224685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.224654    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mps5\" (UniqueName: \"kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.224804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.224793    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.224926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.224861    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.224926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.224905    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.225781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.225759    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.239538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.239507    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.239538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.239523    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.286925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.286854    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mps5\" (UniqueName: \"kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5\") pod \"distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") " pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.353382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.353357    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:27.749582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:27.749540    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"]
Nov 03 03:23:27.751021 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:23:27.750989    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b35c0e4_d1a3_4528_924b_1f095099ee53.slice/crio-73cda1da7bc970ea44542a33dcb64a3335889c92af4b36554aadd5a7088a8bf3 WatchSource:0}: Error finding container 73cda1da7bc970ea44542a33dcb64a3335889c92af4b36554aadd5a7088a8bf3: Status 404 returned error can't find the container with id 73cda1da7bc970ea44542a33dcb64a3335889c92af4b36554aadd5a7088a8bf3
Nov 03 03:23:28.510004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:28.509963    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" event={"ID":"1b35c0e4-d1a3-4528-924b-1f095099ee53","Type":"ContainerStarted","Data":"ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17"}
Nov 03 03:23:28.510004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:28.510009    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" event={"ID":"1b35c0e4-d1a3-4528-924b-1f095099ee53","Type":"ContainerStarted","Data":"73cda1da7bc970ea44542a33dcb64a3335889c92af4b36554aadd5a7088a8bf3"}
Nov 03 03:23:28.628352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:28.628298    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podStartSLOduration=2.628281295 podStartE2EDuration="2.628281295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:23:28.62758395 +0000 UTC m=+1299.858476367" watchObservedRunningTime="2026-11-03 03:23:28.628281295 +0000 UTC m=+1299.859173714"
Nov 03 03:23:29.353542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:29.353504    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:30.030525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:30.030491    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:34.929383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:34.929339    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:35.006977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.006945    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"]
Nov 03 03:23:35.010921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.010905    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.130858 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.126474    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"]
Nov 03 03:23:35.132493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.132460    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"]
Nov 03 03:23:35.162953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.162914    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.162974    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163160    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163234    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163269    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163304    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163333    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163400    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvh6\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.163493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.163429    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.164597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.164566    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:35.301129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.301038    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.301412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.301392    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.301655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.301635    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.301815    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.318686    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.318734    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.318768    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.318846    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvh6\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.318877    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.314761    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.319542    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.314979    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.319822    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.318538    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.320447    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.327572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.320628    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.381039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.381012    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.382038 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.382019    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvh6\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:35.624802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:35.624770    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:36.059154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.059120    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"]
Nov 03 03:23:36.060801 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:23:36.060779    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ed1963_4270_4551_a719_175e5ca80aa7.slice/crio-594bbf825e2f76e5cfbc86306e25249e9d2b9df71fd2c57a4125d358d1815b30 WatchSource:0}: Error finding container 594bbf825e2f76e5cfbc86306e25249e9d2b9df71fd2c57a4125d358d1815b30: Status 404 returned error can't find the container with id 594bbf825e2f76e5cfbc86306e25249e9d2b9df71fd2c57a4125d358d1815b30
Nov 03 03:23:36.063137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.063106    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:23:36.063204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.063173    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:23:36.063242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.063204    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:23:36.557542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.557506    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" event={"ID":"e7ed1963-4270-4551-a719-175e5ca80aa7","Type":"ContainerStarted","Data":"a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977"}
Nov 03 03:23:36.557743 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.557581    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" event={"ID":"e7ed1963-4270-4551-a719-175e5ca80aa7","Type":"ContainerStarted","Data":"594bbf825e2f76e5cfbc86306e25249e9d2b9df71fd2c57a4125d358d1815b30"}
Nov 03 03:23:36.625227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.625190    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:36.685574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:36.685097    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podStartSLOduration=2.685077201 podStartE2EDuration="2.685077201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:23:36.683538358 +0000 UTC m=+1307.914430777" watchObservedRunningTime="2026-11-03 03:23:36.685077201 +0000 UTC m=+1307.915969622"
Nov 03 03:23:37.353870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:37.353842    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:37.391541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:37.391510    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:37.605133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:37.605047    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:23:37.627407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:37.627377    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 03:23:37.627617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:37.627434    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 03:23:38.625054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:38.625020    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 03:23:38.625444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:38.625071    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 03:23:39.625468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:39.625432    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 03:23:39.625870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:39.625488    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 03:23:39.829067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:39.829031    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:40.625662 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:40.625632    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 03:23:40.626152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:40.626119    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 03:23:41.626052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:41.626019    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 03:23:41.626507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:41.626079    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 03:23:41.634871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:41.634845    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:42.625211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:42.625181    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 03:23:42.625386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:42.625229    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 03:23:43.629978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:43.629953    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:43.630353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:43.630215    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:43.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:43.630998    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:23:44.752346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:44.752312    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:49.812439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:49.812395    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:50.178817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.178775    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv" podUID="54e952bf-fd36-44c3-b97c-a1d40130b4f8" containerName="main" containerID="cri-o://de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87" gracePeriod=45
Nov 03 03:23:50.447362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.447338    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:23:50.546143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546108    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir\") pod \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") "
Nov 03 03:23:50.546313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546162    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache\") pod \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") "
Nov 03 03:23:50.546313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546223    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home\") pod \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") "
Nov 03 03:23:50.546386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546360    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm\") pod \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") "
Nov 03 03:23:50.546452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546424    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir" pod "54e952bf-fd36-44c3-b97c-a1d40130b4f8" (UID: "54e952bf-fd36-44c3-b97c-a1d40130b4f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:23:50.546639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546456    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs\") pod \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") "
Nov 03 03:23:50.546639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546485    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8\" (UniqueName: \"kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8\") pod \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\" (UID: \"54e952bf-fd36-44c3-b97c-a1d40130b4f8\") "
Nov 03 03:23:50.546639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546488    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache" pod "54e952bf-fd36-44c3-b97c-a1d40130b4f8" (UID: "54e952bf-fd36-44c3-b97c-a1d40130b4f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:23:50.546639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546591    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home" pod "54e952bf-fd36-44c3-b97c-a1d40130b4f8" (UID: "54e952bf-fd36-44c3-b97c-a1d40130b4f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:23:50.546984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546964    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:23:50.546984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546984    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:23:50.547321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.546993    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:23:50.548825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.548801    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs" pod "54e952bf-fd36-44c3-b97c-a1d40130b4f8" (UID: "54e952bf-fd36-44c3-b97c-a1d40130b4f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:23:50.548902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.548827    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm" pod "54e952bf-fd36-44c3-b97c-a1d40130b4f8" (UID: "54e952bf-fd36-44c3-b97c-a1d40130b4f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:23:50.548902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.548890    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8" pod "54e952bf-fd36-44c3-b97c-a1d40130b4f8" (UID: "54e952bf-fd36-44c3-b97c-a1d40130b4f8"). InnerVolumeSpecName "kube-api-access-7srk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:23:50.648225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.648193    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54e952bf-fd36-44c3-b97c-a1d40130b4f8-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:23:50.648225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.648222    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srk8\" (UniqueName: \"kubernetes.io/projected/54e952bf-fd36-44c3-b97c-a1d40130b4f8-kube-api-access-7srk8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:23:50.648225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.648233    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54e952bf-fd36-44c3-b97c-a1d40130b4f8-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:23:50.668813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.668784    5515 generic.go:356] "Generic (PLEG): container finished" podID="54e952bf-fd36-44c3-b97c-a1d40130b4f8" containerID="de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87" exitCode=0
Nov 03 03:23:50.668970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.668823    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"
Nov 03 03:23:50.668970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.668872    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv" event={"ID":"54e952bf-fd36-44c3-b97c-a1d40130b4f8","Type":"ContainerDied","Data":"de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87"}
Nov 03 03:23:50.668970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.668912    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv" event={"ID":"54e952bf-fd36-44c3-b97c-a1d40130b4f8","Type":"ContainerDied","Data":"5b911a5e7c7b6b1c53421c756dc83fbe64cbf3e632cfadb21dc31b5a6c3e0231"}
Nov 03 03:23:50.668970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.668930    5515 scope.go:122] "RemoveContainer" containerID="de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87"
Nov 03 03:23:50.679369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.679349    5515 scope.go:122] "RemoveContainer" containerID="de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87"
Nov 03 03:23:50.679687 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:23:50.679669    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87\": container with ID starting with de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87 not found: ID does not exist" containerID="de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87"
Nov 03 03:23:50.679767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.679700    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87"} err="failed to get container status \"de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87\": rpc error: code = NotFound desc = could not find container \"de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87\": container with ID starting with de363deef7aee625eea04c006d40c89de16f50886246275d1b38712547271f87 not found: ID does not exist"
Nov 03 03:23:50.836061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.836017    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"]
Nov 03 03:23:50.841534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.841506    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/e2e-conflict-b-e5f308a1-kserve-5fb57d5bd4-c9cbv"]
Nov 03 03:23:50.893387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:50.893356    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e952bf-fd36-44c3-b97c-a1d40130b4f8" path="/var/lib/kubelet/pods/54e952bf-fd36-44c3-b97c-a1d40130b4f8/volumes"
Nov 03 03:23:50.984346 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:23:50.984267    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:23:51.039915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:51.039879    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:23:54.637673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:54.637640    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:55.077817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:55.077733    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:23:59.538794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:23:59.538759    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:04.221392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:04.221353    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:04.431691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:04.431662    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:04.709321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:04.709289    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:04.852742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:04.852705    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:09.344711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:09.344678    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:14.251196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:14.251154    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:20.794516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:20.794487    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:20.984493 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:24:20.984455    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:24:23.993190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:23.993104    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:28.836651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:28.836609    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:32.941866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:32.941832    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:33.085032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:33.084999    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:33.305593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:33.305493    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:33.676331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:33.676278    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:35.485927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.485887    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"]
Nov 03 03:24:35.486796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.486772    5515 cpu_manager.go:414] "containerMap: removing container" podUID="54e952bf-fd36-44c3-b97c-a1d40130b4f8" containerName="main"
Nov 03 03:24:35.486922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.486798    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e952bf-fd36-44c3-b97c-a1d40130b4f8" containerName="main"
Nov 03 03:24:35.486985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.486939    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e952bf-fd36-44c3-b97c-a1d40130b4f8" containerName="main"
Nov 03 03:24:35.490906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.490879    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.534945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.534909    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"]
Nov 03 03:24:35.538611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.538590    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.624740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.624678    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.625107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.625082    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.625312 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.625293    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.625492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.625475    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.625708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.625689    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6l8m\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.625894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.625877    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.626062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626047    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.626232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626215    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.626380 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626364    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.626526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626511    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.626705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626688    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.626895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626872    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.627026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626905    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.627026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626946    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.627026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.626988    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.627026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.627016    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7hj9\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.627250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.627056    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.627250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.627082    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.639569 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.639523    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"]
Nov 03 03:24:35.737581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.737454    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.737843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.737809    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7hj9\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.738006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.737989    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.738114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738101    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.738262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738248    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.738371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738353    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.738494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738481    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.738619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738604    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.738753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738365    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.738833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738747    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6l8m\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.738915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.738890    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.739026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739012    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.739135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739123    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.739247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739233    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.739325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739301    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.739407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739394    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.739507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739494    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.739627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739612    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.739765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739751    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.739865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739853    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.739976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739964    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.740252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739066    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.740529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.740506    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.740649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.740520    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.740741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739576    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.740972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.739804    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.741195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.741170    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.741728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.741702    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.742284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.742261    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"]
Nov 03 03:24:35.744762 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.744737    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.744933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.744912    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.747787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.747738    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.750337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.750317    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.855166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.855085    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.855166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.855087    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7hj9\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:35.856160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.856109    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6l8m\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:35.856815 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:35.856768    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token\") pod \"e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") " pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:36.129852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.129813    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:36.152011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.151988    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:36.741377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.741354    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"]
Nov 03 03:24:36.742482 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:24:36.742454    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442db502_38c6_4dfa_b9e3_1e03b0461ee0.slice/crio-4dcf51a82ebe8718f198884b0cb478e5687dbc3b9c51bf44aac9f222d171db4d WatchSource:0}: Error finding container 4dcf51a82ebe8718f198884b0cb478e5687dbc3b9c51bf44aac9f222d171db4d: Status 404 returned error can't find the container with id 4dcf51a82ebe8718f198884b0cb478e5687dbc3b9c51bf44aac9f222d171db4d
Nov 03 03:24:36.743030 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:24:36.743008    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6388c0bd_7d55_4dd5_b203_ab82a6102a25.slice/crio-9a89e13fb49932f5266b399467716218afc0535709f2cfd49f090b62b903ff53 WatchSource:0}: Error finding container 9a89e13fb49932f5266b399467716218afc0535709f2cfd49f090b62b903ff53: Status 404 returned error can't find the container with id 9a89e13fb49932f5266b399467716218afc0535709f2cfd49f090b62b903ff53
Nov 03 03:24:36.744982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.744949    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:36.745035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.745019    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:36.745126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.745061    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:36.745652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.745624    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:36.745714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.745684    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:36.745770 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.745719    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:36.792160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:36.792134    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"]
Nov 03 03:24:37.011186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.011095    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" event={"ID":"6388c0bd-7d55-4dd5-b203-ab82a6102a25","Type":"ContainerStarted","Data":"ec88d68992a90524c670f18793d1ada4eb8b7e71d2a35b6573ef3d40bbb58476"}
Nov 03 03:24:37.011186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.011154    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" event={"ID":"6388c0bd-7d55-4dd5-b203-ab82a6102a25","Type":"ContainerStarted","Data":"9a89e13fb49932f5266b399467716218afc0535709f2cfd49f090b62b903ff53"}
Nov 03 03:24:37.013153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.013124    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" event={"ID":"442db502-38c6-4dfa-b9e3-1e03b0461ee0","Type":"ContainerStarted","Data":"7e79f283a52fdb436935e49eb595d023e16b76035b04319fafaaf8b1d8cc1062"}
Nov 03 03:24:37.013295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.013176    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" event={"ID":"442db502-38c6-4dfa-b9e3-1e03b0461ee0","Type":"ContainerStarted","Data":"4dcf51a82ebe8718f198884b0cb478e5687dbc3b9c51bf44aac9f222d171db4d"}
Nov 03 03:24:37.130434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.130407    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:37.132175 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.132144    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:37.132289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.132203    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:37.135928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.135884    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podStartSLOduration=2.135868581 podStartE2EDuration="2.135868581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:37.135741277 +0000 UTC m=+1368.366633695" watchObservedRunningTime="2026-11-03 03:24:37.135868581 +0000 UTC m=+1368.366761001"
Nov 03 03:24:37.152804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.152782    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:37.154432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.154404    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:37.154591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.154449    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:37.261166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:37.261102    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podStartSLOduration=2.261086821 podStartE2EDuration="2.261086821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:37.25813024 +0000 UTC m=+1368.489022659" watchObservedRunningTime="2026-11-03 03:24:37.261086821 +0000 UTC m=+1368.491979238"
Nov 03 03:24:38.130927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:38.130892    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:38.131318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:38.130945    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:38.153253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:38.153227    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:38.153357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:38.153273    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:38.921610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:38.921572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:39.136716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:39.135749    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:39.136716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:39.135803    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:39.137289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:39.137172    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:39.156581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:39.154909    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:39.156581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:39.154969    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:40.130890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:40.130848    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:40.131043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:40.130914    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:40.152390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:40.152360    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:40.152819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:40.152490    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:41.130610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.130572    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:41.130794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.130634    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:41.152591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.152564    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:41.152945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.152621    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:41.852265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.852227    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"]
Nov 03 03:24:41.856345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.856327    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.932030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.931717    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"]
Nov 03 03:24:41.932992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.932966    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933184 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933170    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933374    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933703    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cz7\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933814    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933900    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933920    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.933990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.933952    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:41.934046 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:41.934035    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.035344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.035309    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.035668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.035652    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.035777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.035767    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.035863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.035834    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.035917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.035857    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036001    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036067    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cz7\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036157    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036271    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036338    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036123    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.036886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.036769    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.037313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.037289    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.037608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.037562    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.038962 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.038945    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.039326 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.039311    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.116334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.116294    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cz7\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.117042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.117020    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.131282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.131249    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:42.131419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.131327    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:42.152948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.152913    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:42.153282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.152967    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:42.230426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.229779    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:42.716693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.716660    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"]
Nov 03 03:24:42.720703 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:24:42.720670    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7102b72_a704_44f6_a6a4_dbb727d47ed1.slice/crio-9ea59ca60eefdfc12e91c3b1c046b01f04c9286f9f8924f0469bbe7b4576ad58 WatchSource:0}: Error finding container 9ea59ca60eefdfc12e91c3b1c046b01f04c9286f9f8924f0469bbe7b4576ad58: Status 404 returned error can't find the container with id 9ea59ca60eefdfc12e91c3b1c046b01f04c9286f9f8924f0469bbe7b4576ad58
Nov 03 03:24:42.723067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.723036    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:42.723115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.723103    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:42.723146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:42.723134    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:24:43.130613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.130570    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 03:24:43.130795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.130631    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 03:24:43.152372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.152340    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:43.152587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.152404    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:43.159897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.159863    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" event={"ID":"b7102b72-a704-44f6-a6a4-dbb727d47ed1","Type":"ContainerStarted","Data":"3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a"}
Nov 03 03:24:43.160259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.159923    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" event={"ID":"b7102b72-a704-44f6-a6a4-dbb727d47ed1","Type":"ContainerStarted","Data":"9ea59ca60eefdfc12e91c3b1c046b01f04c9286f9f8924f0469bbe7b4576ad58"}
Nov 03 03:24:43.230036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.229984    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:43.231674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.231639    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:43.231807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.231687    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:43.283534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.283472    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podStartSLOduration=2.283457319 podStartE2EDuration="2.283457319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:24:43.28096661 +0000 UTC m=+1374.511859051" watchObservedRunningTime="2026-11-03 03:24:43.283457319 +0000 UTC m=+1374.514349736"
Nov 03 03:24:43.988715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:43.977538    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:44.153102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.152942    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 03:24:44.153102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.153008    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 03:24:44.230247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.230208    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:44.230653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.230270    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:44.636400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.634847    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:44.636400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.636257    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:44.637799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:44.637776    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:24:45.157340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:45.157310    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:45.157618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:45.157600    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:45.158435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:45.158421    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:24:45.230394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:45.230367    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:45.230856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:45.230415    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:46.230319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:46.230283    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:46.230483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:46.230338    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:47.230802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:47.230764    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:47.231283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:47.230830    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:48.231161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:48.231127    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:48.231671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:48.231186    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:49.033569 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:49.033500    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:49.230851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:49.230818    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:24:49.231112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:49.230869    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:24:50.236939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:50.236910    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:50.237457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:50.237240    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:50.237979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:50.237963    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:24:50.984013 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:24:50.983977    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:24:53.922087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:53.922032    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:54.369240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:54.369212    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:24:55.418280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:55.418241    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:24:56.718575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:56.703374    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:56.817697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:56.817646    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:24:56.850724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:56.850674    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Nov 03 03:24:57.223597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.223530    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:57.232816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.232764    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:57.294736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.294679    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:57.569258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.569162    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:57.599668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.599624    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:24:57.659716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.659676    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:24:57.659973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.659942    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:57.661945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.661912    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:57.722575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.704058    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 03 03:24:57.722575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:57.704477    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:24:58.237698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.237653    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 03 03:24:58.238229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.237676    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:58.253134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.238848    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:24:58.253134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.242024    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:24:58.329886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.329837    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:58.331252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.330361    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:24:58.459575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.430843    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:58.541851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.541763    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:24:58.550293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.550272    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:24:58.553669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.553643    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:24:58.579965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.579923    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:24:58.580975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.580954    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:24:58.611567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.601232    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:24:58.611567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.602095    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:24:58.611567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.602194    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:24:58.626933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.626905    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:24:58.632952 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:58.632872    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:24:59.057372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:24:59.057334    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:03.890493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:03.890452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:08.831480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:08.831450    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:13.898336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:13.898289    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:14.554707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.554649    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:14.604243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.604200    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 03 03:25:14.604996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.604511    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:14.605419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.604695    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:14.763578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.739961    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:25:14.824654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.801189    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:25:14.824654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.801264    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:14.824654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:14.820649    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:15.480529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:15.479785    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:25:15.480529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:15.479861    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:18.868007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:18.867972    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:19.653919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:19.653878    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:20.986138 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:25:20.986104    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:25:21.276849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.276762    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"]
Nov 03 03:25:21.282031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.282005    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.335385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.335352    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:25:21.386177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.386145    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\""
Nov 03 03:25:21.388717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.388692    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"]
Nov 03 03:25:21.395843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.395818    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5w6\" (UniqueName: \"kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.395926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.395876    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.396029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.396016    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.496723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.496687    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.496937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.496812    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5w6\" (UniqueName: \"kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.496937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.496847    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.497497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.497474    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.499252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.499230    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.559084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.558999    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5w6\" (UniqueName: \"kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6\") pod \"maas-api-revoke-sub-f6326d17-vntn7\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.604006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.603973    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:21.980757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.980732    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"]
Nov 03 03:25:21.982871 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:25:21.982842    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b7306_df66_41a3_8b26_207d2eab763e.slice/crio-99e8a37ec37672066c09adcf7b0a009dde4090310b743cdf4f0b97dbe2bc75a8 WatchSource:0}: Error finding container 99e8a37ec37672066c09adcf7b0a009dde4090310b743cdf4f0b97dbe2bc75a8: Status 404 returned error can't find the container with id 99e8a37ec37672066c09adcf7b0a009dde4090310b743cdf4f0b97dbe2bc75a8
Nov 03 03:25:21.984710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:21.984691    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:25:22.991334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:22.991299    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7" event={"ID":"202b7306-df66-41a3-8b26-207d2eab763e","Type":"ContainerStarted","Data":"99e8a37ec37672066c09adcf7b0a009dde4090310b743cdf4f0b97dbe2bc75a8"}
Nov 03 03:25:23.769702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:23.769663    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:26.011956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:26.011920    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7" event={"ID":"202b7306-df66-41a3-8b26-207d2eab763e","Type":"ContainerStarted","Data":"ec9bf85bbfc23df7f1fdd025046f4dd908017ea182c4ac036165c9c1f2a2719b"}
Nov 03 03:25:26.132081 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:26.132021    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7" podStartSLOduration=1.427383709 podStartE2EDuration="5.132005375s" totalImagesPullingTime="3.704621666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:26.129796961 +0000 UTC m=+1417.360689380" watchObservedRunningTime="2026-11-03 03:25:26.132005375 +0000 UTC m=+1417.362897792"
Nov 03 03:25:27.019178 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.019139    5515 generic.go:356] "Generic (PLEG): container finished" podID="202b7306-df66-41a3-8b26-207d2eab763e" containerID="ec9bf85bbfc23df7f1fdd025046f4dd908017ea182c4ac036165c9c1f2a2719b" exitCode=0
Nov 03 03:25:27.019607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.019218    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7" event={"ID":"202b7306-df66-41a3-8b26-207d2eab763e","Type":"ContainerDied","Data":"ec9bf85bbfc23df7f1fdd025046f4dd908017ea182c4ac036165c9c1f2a2719b"}
Nov 03 03:25:27.216494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.216463    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"]
Nov 03 03:25:27.220522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.220505    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.282742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.282650    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"]
Nov 03 03:25:27.369781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.369748    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.369940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.369794    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.369982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.369946    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.370017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.369979    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.370050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.370022    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.370149 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.370133    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.370185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.370156    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.370238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.370224    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.370270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.370246    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj7rp\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471382    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471429    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471480    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471525    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471581    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471655    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471680    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj7rp\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471776    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471829    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.471990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.471910    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.472155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.472010    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.472155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.472078    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.472348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.472324    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.472470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.472365    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.474098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.474077    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.474514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.474493    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.535729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.535659    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.538590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.538543    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj7rp\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.548203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.548172    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"]
Nov 03 03:25:27.553382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.553360    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.615024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.614923    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"]
Nov 03 03:25:27.674332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674302    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674341    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674413    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674434    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674531    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674579    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674606    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674683    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r77d\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.674812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.674789    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775699    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775740    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775785    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775809    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775834    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775866    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775901    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.775971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.775956    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r77d\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.776365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.776058    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.776365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.776258    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.776783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.776698    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.776783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.776743    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.776783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.776840    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.777105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.776898    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.778351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.778331    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.778450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.778336    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.834200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.834116    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:27.837946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.837917    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.838081 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.838015    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r77d\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:27.871261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:27.871228    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:28.167127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.167103    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:28.267256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.267219    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"]
Nov 03 03:25:28.282228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.282195    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca\") pod \"202b7306-df66-41a3-8b26-207d2eab763e\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") "
Nov 03 03:25:28.282419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.282237    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6\" (UniqueName: \"kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6\") pod \"202b7306-df66-41a3-8b26-207d2eab763e\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") "
Nov 03 03:25:28.282419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.282267    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca\") pod \"202b7306-df66-41a3-8b26-207d2eab763e\" (UID: \"202b7306-df66-41a3-8b26-207d2eab763e\") "
Nov 03 03:25:28.282669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.282645    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca" pod "202b7306-df66-41a3-8b26-207d2eab763e" (UID: "202b7306-df66-41a3-8b26-207d2eab763e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:25:28.282902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.282876    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-openshift-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:25:28.285017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.284988    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca" pod "202b7306-df66-41a3-8b26-207d2eab763e" (UID: "202b7306-df66-41a3-8b26-207d2eab763e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:25:28.285129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.285058    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6" pod "202b7306-df66-41a3-8b26-207d2eab763e" (UID: "202b7306-df66-41a3-8b26-207d2eab763e"). InnerVolumeSpecName "kube-api-access-sw5w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:25:28.307972 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:25:28.306067    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1709f66c_58e9_4bb3_a37f_6fe85ecc7b67.slice/crio-891a82c4a462bbd43272ccbe6928c7ebe7a57344b980e658683c21cefd0d841f WatchSource:0}: Error finding container 891a82c4a462bbd43272ccbe6928c7ebe7a57344b980e658683c21cefd0d841f: Status 404 returned error can't find the container with id 891a82c4a462bbd43272ccbe6928c7ebe7a57344b980e658683c21cefd0d841f
Nov 03 03:25:28.309878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.309138    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:28.309878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.309230    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:28.309878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.309283    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:28.321054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.320644    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"]
Nov 03 03:25:28.322771 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:25:28.322749    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297031fd_1985_46c3_a747_64214ae62e96.slice/crio-1fe3ea4e81c5f257ef854faa5bab251a851c893fbe20ba57ead4b1fb2d149475 WatchSource:0}: Error finding container 1fe3ea4e81c5f257ef854faa5bab251a851c893fbe20ba57ead4b1fb2d149475: Status 404 returned error can't find the container with id 1fe3ea4e81c5f257ef854faa5bab251a851c893fbe20ba57ead4b1fb2d149475
Nov 03 03:25:28.325646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.325593    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:28.325756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.325674    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:28.325756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.325718    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:28.383728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.383699    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5w6\" (UniqueName: \"kubernetes.io/projected/202b7306-df66-41a3-8b26-207d2eab763e-kube-api-access-sw5w6\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:25:28.383828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.383728    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b7306-df66-41a3-8b26-207d2eab763e-maas-api-xks-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:25:28.737195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:28.737156    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:29.033890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.033850    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" event={"ID":"297031fd-1985-46c3-a747-64214ae62e96","Type":"ContainerStarted","Data":"a2c096338f75705e58a3c5711bdd92b59d28fe1161fd0ad6b7448eb98b2531f1"}
Nov 03 03:25:29.034226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.034183    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" event={"ID":"297031fd-1985-46c3-a747-64214ae62e96","Type":"ContainerStarted","Data":"1fe3ea4e81c5f257ef854faa5bab251a851c893fbe20ba57ead4b1fb2d149475"}
Nov 03 03:25:29.035612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.035536    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" event={"ID":"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67","Type":"ContainerStarted","Data":"30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f"}
Nov 03 03:25:29.035697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.035623    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" event={"ID":"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67","Type":"ContainerStarted","Data":"891a82c4a462bbd43272ccbe6928c7ebe7a57344b980e658683c21cefd0d841f"}
Nov 03 03:25:29.036829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.036813    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"
Nov 03 03:25:29.036921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.036814    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7" event={"ID":"202b7306-df66-41a3-8b26-207d2eab763e","Type":"ContainerDied","Data":"99e8a37ec37672066c09adcf7b0a009dde4090310b743cdf4f0b97dbe2bc75a8"}
Nov 03 03:25:29.036964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.036946    5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e8a37ec37672066c09adcf7b0a009dde4090310b743cdf4f0b97dbe2bc75a8"
Nov 03 03:25:29.159477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.159424    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podStartSLOduration=2.159410049 podStartE2EDuration="2.159410049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:29.157351877 +0000 UTC m=+1420.388244322" watchObservedRunningTime="2026-11-03 03:25:29.159410049 +0000 UTC m=+1420.390302467"
Nov 03 03:25:29.354509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.354391    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podStartSLOduration=2.354375868 podStartE2EDuration="2.354375868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:29.351686585 +0000 UTC m=+1420.582579005" watchObservedRunningTime="2026-11-03 03:25:29.354375868 +0000 UTC m=+1420.585268283"
Nov 03 03:25:29.835107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.835075    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:29.836561 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.836522    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 03:25:29.836651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.836595    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 03:25:29.875741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.875710    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:29.899926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.899891    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 03:25:29.900224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:29.900195    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 03:25:30.834851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:30.834812    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 03:25:30.835270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:30.834879    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 03:25:30.872952 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:30.872918    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 03:25:30.873214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:30.873188    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 03:25:31.834479 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:31.834441    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 03:25:31.834692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:31.834498    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 03:25:31.883574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:31.883043    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 03:25:31.883574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:31.883102    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 03:25:32.835381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:32.835346    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 03:25:32.835603 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:32.835399    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 03:25:32.885030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:32.884990    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 03:25:32.885654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:32.885618    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 03:25:33.835304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:33.835262    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 03:25:33.835499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:33.835318    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 03:25:33.873994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:33.873949    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 03:25:33.874277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:33.874251    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 03:25:34.214172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.214130    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:34.349418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.349388    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"]
Nov 03 03:25:34.355793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.355765    5515 cpu_manager.go:414] "containerMap: removing container" podUID="202b7306-df66-41a3-8b26-207d2eab763e" containerName="revoke-keys"
Nov 03 03:25:34.355995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.355981    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202b7306-df66-41a3-8b26-207d2eab763e" containerName="revoke-keys"
Nov 03 03:25:34.356322 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.356305    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="202b7306-df66-41a3-8b26-207d2eab763e" containerName="revoke-keys"
Nov 03 03:25:34.360688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.360664    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.406013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.405977    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"]
Nov 03 03:25:34.436601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.436543    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.436940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.436916    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.437088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.437073    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.437220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.437204    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.437371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.437355    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.437666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.437645    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.437833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.437816    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.437943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.437928    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljnj\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.445573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.438080    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.547237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.547158    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.547504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.547488    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.547646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.547635    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.547785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.547770    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljnj\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.547879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.547869    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.547973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.547963    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.548051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.548042    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.548125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.548116    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.548202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.548193    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.548848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.548829    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.549165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.549150    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.549737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.549719    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.549909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.549885    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.550403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.550283    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.554372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.552176    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.554372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.553455    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.620913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.620304    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.621410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.621382    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljnj\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.702111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.702079    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:34.835032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.834929    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 03:25:34.835032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.835002    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 03:25:34.843063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.843024    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"]
Nov 03 03:25:34.848222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.848192    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:34.876589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.872362    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 03:25:34.876589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.872417    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 03:25:34.903703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.902020    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"openshift-service-ca.crt\""
Nov 03 03:25:34.903703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.902168    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"distinct-2-w4-b1ad4e-kserve-self-signed-certs\""
Nov 03 03:25:34.903703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.902319    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"default-dockercfg-qtmq4\""
Nov 03 03:25:34.903703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.902989    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b1ad4e\"/\"kube-root-ca.crt\""
Nov 03 03:25:34.906268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.906243    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"]
Nov 03 03:25:34.926062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.926022    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:34.954493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.954459    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:34.954731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.954618    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd57m\" (UniqueName: \"kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:34.954731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.954662    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:34.954849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.954777    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:34.954849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.954827    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:34.954986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:34.954971    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.061448    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.061590    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.061691    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd57m\" (UniqueName: \"kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.061725    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.061820    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.061860    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.062367    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.063954    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.066579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.064229    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.075620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.074956    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.085528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.084007    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.128450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.128415    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd57m\" (UniqueName: \"kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m\") pod \"distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") " pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.144282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.144250    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"]
Nov 03 03:25:35.145252 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:25:35.145226    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256bd96e_dc29_439c_9978_a0cc51c869c7.slice/crio-08de1684cfe2db3e960e2f46b6ebcd9734f100ad81366cd889610d897b624ade WatchSource:0}: Error finding container 08de1684cfe2db3e960e2f46b6ebcd9734f100ad81366cd889610d897b624ade: Status 404 returned error can't find the container with id 08de1684cfe2db3e960e2f46b6ebcd9734f100ad81366cd889610d897b624ade
Nov 03 03:25:35.148247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.148208    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:35.148322 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.148298    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:35.148367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.148346    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:35.165891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.165850    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:35.612293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.612257    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"]
Nov 03 03:25:35.621097 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:25:35.621014    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb747ca_ec06_45af_986d_dc4582cd3583.slice/crio-7312c895cd593bc68cc6f780e403634092d0bdbc7727e06bcfd5a3fc5bd260ff WatchSource:0}: Error finding container 7312c895cd593bc68cc6f780e403634092d0bdbc7727e06bcfd5a3fc5bd260ff: Status 404 returned error can't find the container with id 7312c895cd593bc68cc6f780e403634092d0bdbc7727e06bcfd5a3fc5bd260ff
Nov 03 03:25:35.995120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.995024    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:35.995704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.995649    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:35.997529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:35.997487    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:25:36.131410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.131377    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" event={"ID":"8fb747ca-ec06-45af-986d-dc4582cd3583","Type":"ContainerStarted","Data":"3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd"}
Nov 03 03:25:36.131665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.131422    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" event={"ID":"8fb747ca-ec06-45af-986d-dc4582cd3583","Type":"ContainerStarted","Data":"7312c895cd593bc68cc6f780e403634092d0bdbc7727e06bcfd5a3fc5bd260ff"}
Nov 03 03:25:36.133421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.133392    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" event={"ID":"256bd96e-dc29-439c-9978-a0cc51c869c7","Type":"ContainerStarted","Data":"e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b"}
Nov 03 03:25:36.133540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.133433    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" event={"ID":"256bd96e-dc29-439c-9978-a0cc51c869c7","Type":"ContainerStarted","Data":"08de1684cfe2db3e960e2f46b6ebcd9734f100ad81366cd889610d897b624ade"}
Nov 03 03:25:36.165974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.165945    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:36.265197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.265117    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:36.266392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.266350    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:36.268703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.268681    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:25:36.491642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.491570    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podStartSLOduration=2.491533752 podStartE2EDuration="2.491533752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:36.489512276 +0000 UTC m=+1427.720404694" watchObservedRunningTime="2026-11-03 03:25:36.491533752 +0000 UTC m=+1427.722426170"
Nov 03 03:25:36.492147 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.492119    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podStartSLOduration=2.492107685 podStartE2EDuration="2.492107685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:36.370526179 +0000 UTC m=+1427.601418598" watchObservedRunningTime="2026-11-03 03:25:36.492107685 +0000 UTC m=+1427.723000100"
Nov 03 03:25:36.702751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.702715    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:36.704627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.704593    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:36.704740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:36.704655    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:37.633610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:37.633575    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:25:37.702671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:37.702634    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:37.702857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:37.702681    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:38.704017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:38.703979    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:38.704644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:38.704614    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:39.543973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.542653    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:39.553413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.552659    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 03 03:25:39.561362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.560625    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:39.612530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.612467    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 03:25:39.636766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.636708    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:39.637450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.637411    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:25:39.704074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.703937    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:39.704074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.703997    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:39.819757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.819654    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:25:39.928689 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:39.928638    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 03:25:40.380585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.377912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:40.459583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.457685    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.554575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.545656    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.554575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.546456    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:25:40.554575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.553319    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.554575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.553736    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:25:40.566574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.565732    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.566574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.566070    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:25:40.638657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.625889    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.638657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.626223    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:25:40.639659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.639624    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.640204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.640183    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:25:40.641677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.641650    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.656572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.642144    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:25:40.715576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.715000    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:40.715576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.715060    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:40.875577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.830360    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:40.883627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:40.876278    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:25:41.548194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.548151    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:41.555133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.555083    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:41.593496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.593441    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 03:25:41.625601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.611724    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:25:41.625601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.611801    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:25:41.626125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.626049    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:25:41.756586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.756538    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:41.757171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.757142    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:41.758822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.758802    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:25:41.774738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.774702    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:25:41.788718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.788692    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:25:41.797090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.797061    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:25:41.822335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:41.822246    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:25:42.703067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:42.703032    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:42.703271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:42.703104    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:42.911275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:42.911211    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:43.060846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.060764    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:43.661391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.661345    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"]
Nov 03 03:25:43.675152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.675121    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.703500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.703469    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 03:25:43.703697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.703529    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 03:25:43.727538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.727506    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"]
Nov 03 03:25:43.756756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.756696    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:43.775424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.775388    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9902a0c2-openshift-default-dockercfg-q88mb\""
Nov 03 03:25:43.896645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.875880    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.898913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.898887    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.899233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.899210    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.899395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.899380    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n844c\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.899534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.899520    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.899986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.899964    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.900268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.900244    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.900542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.900514    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:43.900782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:43.900763    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.017626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.014613    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.017626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.014725    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.017626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.014798    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.017626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.014823    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.017626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.014854    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.017626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.014906    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n844c\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.034510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.034471    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:44.036061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.036031    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.036421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.036402    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.038634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.038612    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.039127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.037520    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.039246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.038216    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.039321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.037075    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.039321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.039140    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.039413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.039392    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.055976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.055945    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.059006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.058980    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.098707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.098669    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.098882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.098725    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n844c\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c\") pod \"e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") " pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.293876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.293835    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:44.829245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.826023    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:44.829245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.826122    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:44.829245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.826166    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:25:44.841591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.832839    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"]
Nov 03 03:25:44.908586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:44.906930    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" event={"ID":"988a661b-4a24-4a07-8c85-953950bdb3f2","Type":"ContainerStarted","Data":"2d7f3e61824d01feae14d963f01d59de9cef865e482e5a90e14ee51c28095e89"}
Nov 03 03:25:45.166690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.166654    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:45.184907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.184876    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:45.239813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.239775    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:45.240970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.240717    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:45.241349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.241320    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:25:45.633982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.633947    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:45.914490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.914402    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" event={"ID":"988a661b-4a24-4a07-8c85-953950bdb3f2","Type":"ContainerStarted","Data":"d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"}
Nov 03 03:25:45.925917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:45.925894    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:25:46.088079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:46.088024    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podStartSLOduration=3.088008497 podStartE2EDuration="3.088008497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:25:46.03628712 +0000 UTC m=+1437.267179538" watchObservedRunningTime="2026-11-03 03:25:46.088008497 +0000 UTC m=+1437.318900914"
Nov 03 03:25:46.294265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:46.294181    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:46.300214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:46.300188    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:46.925684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:46.925645    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:46.926256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:46.926238    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:25:49.043888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:49.043851    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:49.212993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:49.212953    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:49.360482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:49.360451    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:50.911592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:50.911557    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:51.057499 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:25:51.057459    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:25:52.351423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:52.351388    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:52.531282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:52.531047    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:53.101711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:53.101674    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:53.265759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:53.265727    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:54.418836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:54.418808    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:54.560243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:54.560177    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:54.719627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:54.711034    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:55.832446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:55.832378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:56.073234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:56.073203    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:56.670620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:56.670325    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:56.821822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:56.821792    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:25:56.972839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:25:56.972750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:00.938261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:00.938207    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:04.533577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:04.533080    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:04.567576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:04.567430    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:04.571580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:04.567919    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:04.642577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:04.637944    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:04.765111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:04.763671    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:04.805485 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:04.804099    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.011775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.011728    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.434579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.426678    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:26:05.434579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.426755    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.540400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.538790    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.540400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.539234    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:26:05.684968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.653007    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 03:26:05.684968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.653313    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 03 03:26:05.684968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.654078    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:26:05.684968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.654139    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:26:05.684968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.655076    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.684968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.655350    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:26:05.772382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.770695    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.772382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.771164    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:26:05.772382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.771483    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:26:05.808888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.807044    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:05.808888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.807503    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:26:05.883189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:05.856820    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 03 03:26:06.036725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.020203    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.054575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.040896    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:26:06.189576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.184973    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.377635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.363656    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:26:06.377635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.363721    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.546614 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.542133    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.660852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.660742    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.727665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.662669    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.727665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.662918    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.865452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.779727    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:06.865452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.780094    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 03 03:26:06.865452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.840035    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:26:06.868530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.868495    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 03:26:06.934733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.934641    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Nov 03 03:26:06.976011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:06.975971    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:07.050704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.050660    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.117732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.117687    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:26:07.118019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.117986    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.378869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.357650    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:26:07.378869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.357715    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.543657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.543610    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:26:07.685765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.685663    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.712625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.687635    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 03:26:07.712625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.699634    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.712625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.703678    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.782576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.781648    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.782576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.781682    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:07.865733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.864536    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:26:07.869829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.869025    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 03:26:07.936639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:07.935708    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:26:08.098007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.097975    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:26:08.105988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.104944    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:08.112825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.112802    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:26:08.155876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.155690    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:26:08.212192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.190046    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:26:08.231371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.231347    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:26:08.233186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:08.233162    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:26:12.049145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:12.049113    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:12.399671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:12.399630    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:16.923890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:16.923858    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:18.982068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:18.982030    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:19.689453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.689422    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"]
Nov 03 03:26:19.694721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.694702    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.740244 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.740214    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"]
Nov 03 03:26:19.744324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.744304    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.796573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.796533    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"]
Nov 03 03:26:19.824397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824361    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.824397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824401    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.824688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824423    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.824688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824482    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.824688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824584    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjjd\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.824688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824610    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.824688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824627    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.824688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824658    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.824964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824717    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.824964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824805    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.824964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824859    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.824964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824956    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.825108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.824986    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.825108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.825003    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.825108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.825102    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.825234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.825124    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.825234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.825140    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvwx\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.825234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.825161    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.849264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.849229    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"]
Nov 03 03:26:19.927211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.927177    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.927474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.927452    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.927635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.927615    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvwx\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.927764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.927747    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.927928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.927911    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.928050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928034    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.928181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928162    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.928312 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928287    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.928465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928341    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjjd\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.928465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928365    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.928465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928390    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.928465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928430    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.928465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928443    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928482    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928519    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928590    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928727    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928792    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928820    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.929081    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.929195    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.929434    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.929453    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.929921    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.930038    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.930109    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.928088    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.930913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.930668    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.933313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.933285    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.933468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.933443    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.935256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.935231    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.935504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.935481    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.992692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.992606    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:19.993075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.992751    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjjd\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.993075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.992750    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:19.993075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:19.992841    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvwx\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:20.008377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.008351    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:20.058038 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.058002    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:20.443646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.443609    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"]
Nov 03 03:26:20.444127 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:26:20.444100    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095440b1_e9d8_4acc_90c5_d973820d82d2.slice/crio-1d450f2c9c2000fd7fd72a2b7905dafa9eee56bd8bd2d6adbcafc2024660a410 WatchSource:0}: Error finding container 1d450f2c9c2000fd7fd72a2b7905dafa9eee56bd8bd2d6adbcafc2024660a410: Status 404 returned error can't find the container with id 1d450f2c9c2000fd7fd72a2b7905dafa9eee56bd8bd2d6adbcafc2024660a410
Nov 03 03:26:20.446527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.446494    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:20.446620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.446581    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:20.446620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.446615    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:20.497508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.497479    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"]
Nov 03 03:26:20.498536 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:26:20.498508    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bb2648_08d7_436b_82fa_8a06538a04ae.slice/crio-1d33c8b0a78fd1863be58c280f030377248f76dda217d52999c9ff1c350b3123 WatchSource:0}: Error finding container 1d33c8b0a78fd1863be58c280f030377248f76dda217d52999c9ff1c350b3123: Status 404 returned error can't find the container with id 1d33c8b0a78fd1863be58c280f030377248f76dda217d52999c9ff1c350b3123
Nov 03 03:26:20.501228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.501199    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:20.501278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.501265    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:20.501313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.501294    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:20.518721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.518685    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" event={"ID":"095440b1-e9d8-4acc-90c5-d973820d82d2","Type":"ContainerStarted","Data":"1d450f2c9c2000fd7fd72a2b7905dafa9eee56bd8bd2d6adbcafc2024660a410"}
Nov 03 03:26:20.519992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:20.519971    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" event={"ID":"71bb2648-08d7-436b-82fa-8a06538a04ae","Type":"ContainerStarted","Data":"1d33c8b0a78fd1863be58c280f030377248f76dda217d52999c9ff1c350b3123"}
Nov 03 03:26:21.009911 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:26:21.009863    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:26:21.527438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:21.527398    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" event={"ID":"095440b1-e9d8-4acc-90c5-d973820d82d2","Type":"ContainerStarted","Data":"1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1"}
Nov 03 03:26:21.529404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:21.529378    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" event={"ID":"71bb2648-08d7-436b-82fa-8a06538a04ae","Type":"ContainerStarted","Data":"f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7"}
Nov 03 03:26:21.723486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:21.714296    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podStartSLOduration=2.714276078 podStartE2EDuration="2.714276078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:21.712814288 +0000 UTC m=+1472.943706707" watchObservedRunningTime="2026-11-03 03:26:21.714276078 +0000 UTC m=+1472.945168497"
Nov 03 03:26:21.843624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:21.842081    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podStartSLOduration=2.842066205 podStartE2EDuration="2.842066205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:21.840656697 +0000 UTC m=+1473.071549114" watchObservedRunningTime="2026-11-03 03:26:21.842066205 +0000 UTC m=+1473.072958623"
Nov 03 03:26:21.990600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:21.985428    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:22.021574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.016624    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:22.021574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.016774    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 03:26:22.021574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.016832    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 03:26:22.061577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.059017    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:22.098637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.090598    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 03:26:22.098637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.090653    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 03:26:22.194575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:22.170461    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:23.012585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:23.011917    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 03:26:23.012585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:23.012010    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 03:26:23.058775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:23.058743    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 03:26:23.059165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:23.058805    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 03:26:24.012571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:24.012043    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 03:26:24.012571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:24.012140    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 03:26:24.058732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:24.058699    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 03:26:24.058905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:24.058753    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 03:26:25.049571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:25.047725    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 03:26:25.049571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:25.047806    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 03:26:25.060618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:25.058641    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 03:26:25.060618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:25.058689    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 03:26:26.011979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:26.011939    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 03:26:26.012699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:26.012287    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 03:26:26.058978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:26.058947    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 03:26:26.059239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:26.059212    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 03:26:27.009470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:27.009432    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 03:26:27.009863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:27.009496    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 03:26:27.055844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:27.055813    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:27.058875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:27.058853    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 03:26:27.058990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:27.058899    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 03:26:28.028095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.028053    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"]
Nov 03 03:26:28.033309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.033272    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.059620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.059585    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.059878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.059844    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:28.060045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.059847    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060153    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060197    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060258    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060311    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060362    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060396    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060472    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsvw\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.060673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.060654    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:28.061530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.061513    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:26:28.135922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.135889    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"]
Nov 03 03:26:28.161624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.161583    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsvw\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.161874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.161737    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.161874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.161771    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.161874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.161845    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.161874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.161874    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.162231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.161964    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.162231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.162037    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.162231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.162089    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.162231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.162125    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.162538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.162381    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.163353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.163321    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.163629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.163602    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.163841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.163796    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.164012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.163991    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.172147 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.172118    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.173543 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.173518    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.189371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.189343    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:28.189679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.189658    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:28.190657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.190637    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:26:28.223898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.223859    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsvw\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.224454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.224436    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.354343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.354268    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:28.790361 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:26:28.790316    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046e4002_87ed_4a9b_a4a1_de4af35a9845.slice/crio-2c7a5986f76c9fcfddaa3f18f1d02ae6ca14db7931657acfd8829d76d3d4b7bc WatchSource:0}: Error finding container 2c7a5986f76c9fcfddaa3f18f1d02ae6ca14db7931657acfd8829d76d3d4b7bc: Status 404 returned error can't find the container with id 2c7a5986f76c9fcfddaa3f18f1d02ae6ca14db7931657acfd8829d76d3d4b7bc
Nov 03 03:26:28.790490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.790442    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"]
Nov 03 03:26:28.792890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.792859    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:28.792958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.792920    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:28.792958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:28.792948    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:29.595293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:29.595253    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" event={"ID":"046e4002-87ed-4a9b-a4a1-de4af35a9845","Type":"ContainerStarted","Data":"4ca1d5e6e47155ce4880c5147cfefdadd0b867f4483cf51578c593e4deba430a"}
Nov 03 03:26:29.595750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:29.595317    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" event={"ID":"046e4002-87ed-4a9b-a4a1-de4af35a9845","Type":"ContainerStarted","Data":"2c7a5986f76c9fcfddaa3f18f1d02ae6ca14db7931657acfd8829d76d3d4b7bc"}
Nov 03 03:26:29.788576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:29.787690    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podStartSLOduration=2.787671037 podStartE2EDuration="2.787671037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:29.776842244 +0000 UTC m=+1481.007734663" watchObservedRunningTime="2026-11-03 03:26:29.787671037 +0000 UTC m=+1481.018563456"
Nov 03 03:26:30.354715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:30.354674    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:30.356265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:30.356237    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:26:30.356391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:30.356299    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:26:31.355375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:31.355340    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:26:31.355805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:31.355395    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:26:31.934162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:31.934130    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:32.355487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:32.355401    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:26:32.355487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:32.355463    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:26:33.355241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:33.355208    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:26:33.355432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:33.355269    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:26:34.354883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:34.354844    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:26:34.355251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:34.354904    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:26:35.126417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.126383    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:35.355594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.355527    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:26:35.356007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.355626    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:26:35.691017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.690981    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"]
Nov 03 03:26:35.696323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.696303    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.753832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.753796    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d82a79-openshift-default-dockercfg-9sb7s\""
Nov 03 03:26:35.799575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.798734    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:35.867102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.833803    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.934723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.912190    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"]
Nov 03 03:26:35.961069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.960989    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.961426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.961400    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.961764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.961731    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.962154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.962135    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.962580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.962541    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.962988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.962939    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lww\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.963499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.963425    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.963887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.963838    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:35.983586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:35.983481    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:36.122057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.118948    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.130699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.130667    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.130942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.130921    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.131121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.131102    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.131310 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.131291    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.131637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.131614    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.131850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.131830    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lww\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.132032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.132014    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.132284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.132263    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.132927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.127159    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.133412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.133391    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.133826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.133806    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.139011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.138988    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.140114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.140092    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.154255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.140815    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.179912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.179839    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.203573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.197097    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.209583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.208029    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lww\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww\") pod \"e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") " pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.355264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.355170    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:36.543129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.542404    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:36.543662 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.543182    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:36.544304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.544259    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:26:36.837226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.837118    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"]
Nov 03 03:26:36.844018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.843984    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:36.846649 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:26:36.846614    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45285d6d_5238_4608_ad26_49df796027b2.slice/crio-05a3826ec560d37a1e4545c90ac447bf3f82d3171732e4a3336b166fc7906380 WatchSource:0}: Error finding container 05a3826ec560d37a1e4545c90ac447bf3f82d3171732e4a3336b166fc7906380: Status 404 returned error can't find the container with id 05a3826ec560d37a1e4545c90ac447bf3f82d3171732e4a3336b166fc7906380
Nov 03 03:26:36.849163 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.849128    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:36.849250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.849201    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:36.849250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:36.849237    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:37.648826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:37.648789    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" event={"ID":"45285d6d-5238-4608-ad26-49df796027b2","Type":"ContainerStarted","Data":"422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"}
Nov 03 03:26:37.649236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:37.648854    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" event={"ID":"45285d6d-5238-4608-ad26-49df796027b2","Type":"ContainerStarted","Data":"05a3826ec560d37a1e4545c90ac447bf3f82d3171732e4a3336b166fc7906380"}
Nov 03 03:26:37.736486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:37.736449    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:37.774041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:37.773985    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podStartSLOduration=2.773971237 podStartE2EDuration="2.773971237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:37.770912465 +0000 UTC m=+1489.001804875" watchObservedRunningTime="2026-11-03 03:26:37.773971237 +0000 UTC m=+1489.004863696"
Nov 03 03:26:38.356400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:38.356359    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:38.361503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:38.361481    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:38.655092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:38.655056    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:38.656247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:38.656231    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:26:40.817097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:40.817066    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:40.991478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:40.991442    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:41.139152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:41.139121    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:42.021130 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:42.021098    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:42.644201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:42.644168    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:26:44.559652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:44.556273    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:44.793576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:44.785848    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:44.864704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:44.864663    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:45.034715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.030756    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 03 03:26:45.144468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.144374    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:45.182582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.169888    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"]
Nov 03 03:26:45.183469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.183415    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:45.427688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.426424    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"]
Nov 03 03:26:45.427688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.426603    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:45.490576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.450674    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:45.553573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.549738    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:45.553573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.551591    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:45.553573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.551814    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtffx\" (UniqueName: \"kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx\") pod \"payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:45.682705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.682602    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtffx\" (UniqueName: \"kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx\") pod \"payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:46.087633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.077884    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.087633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.077953    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:46.087633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.902131    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 03 03:26:46.087633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.988200    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"]
Nov 03 03:26:46.087633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.083622    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:26:46.087633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.683965    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d82a79-dockercfg-sfxd4\""
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.089713    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.091019    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.052662    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.091389    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.052704    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.052725    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.129199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:45.684026    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d82a79\""
Nov 03 03:26:46.157575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.152750    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.157575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.153185    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:26:46.208850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.183024    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"]
Nov 03 03:26:46.208850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.183586    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.208850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.183663    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.208850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.184019    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:26:46.208850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.186151    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") " pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.208850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.186213    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqv2\" (UniqueName: \"kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2\") pod \"payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") " pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.306586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.302800    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") " pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.306586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.302901    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqv2\" (UniqueName: \"kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2\") pod \"payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") " pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.352622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.352229    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtffx\" (UniqueName: \"kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx\") pod \"payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:46.367258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.367218    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") " pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.465718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.455676    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.465718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.457595    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:26:46.564574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.553750    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqv2\" (UniqueName: \"kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2\") pod \"payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") " pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:46.575610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.554726    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:46.600639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.554938    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:46.615629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.607670    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:26:46.847013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:46.846980    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:47.086914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.084672    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:26:47.086914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.086362    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:47.136577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.092618    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 03 03:26:47.164577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.138754    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:47.224892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.174576    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:26:47.224892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.139327    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:47.224892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.159850    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:47.224892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.176106    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:26:47.224892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.194074    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:47.611190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.592100    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:47.622374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.619892    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:26:47.668385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:47.668341    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.178791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.178745    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.202980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.179650    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.284200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.284156    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.285622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.133712    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.286300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.171800    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:26:48.435383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.352430    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"]
Nov 03 03:26:48.655030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.359129    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:48.655030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.179709    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 03:26:48.655030 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:26:48.505827    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea000d_fcfa_43cd_98d7_acb5f5ed9b18.slice/crio-aacb1d8d23ef63ccd19a444cc76e4e3ad98dc5b719065fcf00981b2541f44a59 WatchSource:0}: Error finding container aacb1d8d23ef63ccd19a444cc76e4e3ad98dc5b719065fcf00981b2541f44a59: Status 404 returned error can't find the container with id aacb1d8d23ef63ccd19a444cc76e4e3ad98dc5b719065fcf00981b2541f44a59
Nov 03 03:26:48.655030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.546444    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.655030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.601072    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:48.655030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.636142    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 03:26:48.687742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:48.685692    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 03:26:49.142765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.142722    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-d45dp" podUID="ef1853a0-c729-463f-ade2-76de6dddca26" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:26:49.224234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.223343    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 03:26:49.339765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.337689    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 03 03:26:49.405610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.378686    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:26:49.450169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.448596    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 03:26:49.543465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.543432    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf" event={"ID":"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18","Type":"ContainerStarted","Data":"aacb1d8d23ef63ccd19a444cc76e4e3ad98dc5b719065fcf00981b2541f44a59"}
Nov 03 03:26:49.543707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.543687    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 03:26:49.543880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.543866    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 03:26:49.544013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.544001    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw"
Nov 03 03:26:49.544151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.544138    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 03:26:49.544274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.544262    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 03:26:49.544470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.544457    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:26:49.545174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.545147    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 03:26:49.545265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.545188    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"]
Nov 03 03:26:49.724722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.724691    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:26:49.728518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.728387    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:26:49.728676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.728563    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:26:49.730357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:49.730283    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:26:50.409153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.409118    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf" event={"ID":"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18","Type":"ContainerStarted","Data":"3f2e491591694884536c3b5201903fbdf7fd91453cdf7a728b9e3cb7049565c2"}
Nov 03 03:26:50.409632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.409184    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:26:50.410951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.410928    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x" event={"ID":"d659c741-776c-4043-a094-2d9e5b653777","Type":"ContainerStarted","Data":"c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8"}
Nov 03 03:26:50.411070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.410970    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x" event={"ID":"d659c741-776c-4043-a094-2d9e5b653777","Type":"ContainerStarted","Data":"5694a34d7811810c5bd06ae27a6b76db2816d65e0e897bf507a2b038f0ab34b3"}
Nov 03 03:26:50.411070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.411037    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:26:50.704093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.703833    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x" podStartSLOduration=6.703818621 podStartE2EDuration="6.703818621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:50.702522711 +0000 UTC m=+1501.933415130" watchObservedRunningTime="2026-11-03 03:26:50.703818621 +0000 UTC m=+1501.934711039"
Nov 03 03:26:50.851582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:50.846347    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf" podStartSLOduration=5.846325552 podStartE2EDuration="5.846325552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:26:50.837375737 +0000 UTC m=+1502.068268157" watchObservedRunningTime="2026-11-03 03:26:50.846325552 +0000 UTC m=+1502.077217972"
Nov 03 03:26:51.116454 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:26:51.116420    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:26:52.940665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:52.940632    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:56.136636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.136604    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:56.454451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.454355    5515 generic.go:356] "Generic (PLEG): container finished" podID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerID="d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb" exitCode=0
Nov 03 03:26:56.454451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.454429    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" event={"ID":"988a661b-4a24-4a07-8c85-953950bdb3f2","Type":"ContainerDied","Data":"d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"}
Nov 03 03:26:56.454926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.454908    5515 scope.go:122] "RemoveContainer" containerID="d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"
Nov 03 03:26:56.456371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.456338    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:56.456477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.456425    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:56.456477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:56.456471    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:26:57.489220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:57.489182    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" event={"ID":"988a661b-4a24-4a07-8c85-953950bdb3f2","Type":"ContainerStarted","Data":"8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d"}
Nov 03 03:26:57.837864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:57.837774    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:26:58.294684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:58.294648    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:26:58.295763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:58.295737    5515 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 03:26:58.295885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:58.295793    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 03:26:59.295212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:59.295176    5515 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 03:26:59.295700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:26:59.295248    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 03:27:00.294710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:00.294673    5515 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 03:27:00.294907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:00.294730    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 03:27:01.295190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:01.295148    5515 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 03:27:01.295732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:01.295223    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 03:27:01.420641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:01.420616    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:27:01.420841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:01.420683    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:27:01.925746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:01.925712    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:27:02.294602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:02.294491    5515 patch_prober.go:29] interesting pod/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 03:27:02.294602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:02.294542    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 03:27:02.755218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:02.755182    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:03.299064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:03.299021    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:27:03.300316 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:03.300298    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:27:05.458703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:05.458669    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:06.162154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:06.162122    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:06.822578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:06.794460    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:07.127682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:07.123698    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:07.918924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:07.918893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:11.309202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:11.309165    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:11.466628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:11.466578    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:11.617600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:11.617571    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:13.421253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:13.421221    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:14.529288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:14.529246    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:14.725574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:14.721676    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:14.968507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:14.965961    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"]
Nov 03 03:27:14.976878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:14.976848    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.023933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.021456    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"]
Nov 03 03:27:15.038151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.038116    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlgd5\" (UniqueName: \"kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5\") pod \"payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.038653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.038627    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.124578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.115440    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b562f5\""
Nov 03 03:27:15.128577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.126740    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b562f5-dockercfg-p2886\""
Nov 03 03:27:15.141253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.137478    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"]
Nov 03 03:27:15.150869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.142498    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.151286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.151255    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlgd5\" (UniqueName: \"kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5\") pod \"payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.181584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.143584    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.181856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.163866    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.205059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.202767    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"]
Nov 03 03:27:15.270058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.269965    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.270378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.270353    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghqn\" (UniqueName: \"kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn\") pod \"payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.286354 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.286292    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlgd5\" (UniqueName: \"kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5\") pod \"payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.322445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.322413    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:15.372285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.372249    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.372482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.372343    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghqn\" (UniqueName: \"kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn\") pod \"payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.373064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.373036    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.435004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.434973    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghqn\" (UniqueName: \"kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn\") pod \"payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.610624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.610591    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:15.708987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.708955    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"]
Nov 03 03:27:15.716143 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:27:15.716110    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93085299_7eb4_495f_ad03_f95594fc176a.slice/crio-463cf80befc5ad3e3754b561a9517cc0003b75cdd98faa2521a12bce3ad5472f WatchSource:0}: Error finding container 463cf80befc5ad3e3754b561a9517cc0003b75cdd98faa2521a12bce3ad5472f: Status 404 returned error can't find the container with id 463cf80befc5ad3e3754b561a9517cc0003b75cdd98faa2521a12bce3ad5472f
Nov 03 03:27:15.990852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:15.990825    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"]
Nov 03 03:27:15.991717 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:27:15.991693    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8d5e33_b51b_4707_8eda_88fa0881d9b6.slice/crio-c5ddd64a8dd6dfc97a681e978e7b831711dd3677243f8e87277749983324f3bb WatchSource:0}: Error finding container c5ddd64a8dd6dfc97a681e978e7b831711dd3677243f8e87277749983324f3bb: Status 404 returned error can't find the container with id c5ddd64a8dd6dfc97a681e978e7b831711dd3677243f8e87277749983324f3bb
Nov 03 03:27:16.715822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.715786    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d" event={"ID":"93085299-7eb4-495f-ad03-f95594fc176a","Type":"ContainerStarted","Data":"0d069af0db603245579925af6d7bf94068d8ea10b18c2475684a924a1efa55dc"}
Nov 03 03:27:16.716375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.715849    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d" event={"ID":"93085299-7eb4-495f-ad03-f95594fc176a","Type":"ContainerStarted","Data":"463cf80befc5ad3e3754b561a9517cc0003b75cdd98faa2521a12bce3ad5472f"}
Nov 03 03:27:16.716375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.715914    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:16.717563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.717519    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g" event={"ID":"6b8d5e33-b51b-4707-8eda-88fa0881d9b6","Type":"ContainerStarted","Data":"f5b7cbf731608d7e02fad39a2c8537611e80bf5e16eba6c2aff61cd90ee12bff"}
Nov 03 03:27:16.717686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.717591    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g" event={"ID":"6b8d5e33-b51b-4707-8eda-88fa0881d9b6","Type":"ContainerStarted","Data":"c5ddd64a8dd6dfc97a681e978e7b831711dd3677243f8e87277749983324f3bb"}
Nov 03 03:27:16.717686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.717623    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:16.835769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.835718    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d" podStartSLOduration=2.835705382 podStartE2EDuration="2.835705382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:16.832890761 +0000 UTC m=+1528.063783192" watchObservedRunningTime="2026-11-03 03:27:16.835705382 +0000 UTC m=+1528.066597800"
Nov 03 03:27:16.985665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:16.981890    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g" podStartSLOduration=1.981865198 podStartE2EDuration="1.981865198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:27:16.963197366 +0000 UTC m=+1528.194089785" watchObservedRunningTime="2026-11-03 03:27:16.981865198 +0000 UTC m=+1528.212757630"
Nov 03 03:27:18.467512 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:18.467471    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:18.631917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:18.631880    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:18.779653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:18.773912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:19.033628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:19.033478    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:20.984421 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:27:20.984382    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:27:23.525079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:23.524994    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:23.682591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:23.682276    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:27.727921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:27.727893    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:27:27.728404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:27.727958    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:27:28.519401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:28.519365    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:33.236619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:33.236585    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:33.416987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:33.416949    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:38.281437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:38.281402    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:43.102353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:43.102319    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:27:43.128463 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:43.128402    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:45.532766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:45.532718    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:27:45.691688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:45.691642    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:27:48.037920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:48.037882    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:49.554058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:49.554028    5515 scope.go:122] "RemoveContainer" containerID="533cc4f5b0d8f09838df5451299258b135629e92cc08dd071106f461d5500bfe"
Nov 03 03:27:50.989477 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:27:50.989438    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:27:52.769441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:52.769408    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:52.936589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:52.936492    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:53.084941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:53.084852    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:54.047181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:54.045529    5515 patch_prober.go:29] interesting pod/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": read tcp 10.134.0.2:48484->10.134.0.83:15021: read: connection reset by peer" start-of-body=
Nov 03 03:27:54.047181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:54.045635    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": read tcp 10.134.0.2:48484->10.134.0.83:15021: read: connection reset by peer"
Nov 03 03:27:54.079163 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:54.079139    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:55.076636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:55.076601    5515 generic.go:356] "Generic (PLEG): container finished" podID="45285d6d-5238-4608-ad26-49df796027b2" containerID="422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954" exitCode=0
Nov 03 03:27:55.077235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:55.076676    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" event={"ID":"45285d6d-5238-4608-ad26-49df796027b2","Type":"ContainerDied","Data":"422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"}
Nov 03 03:27:55.077235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:55.077141    5515 scope.go:122] "RemoveContainer" containerID="422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"
Nov 03 03:27:55.078594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:55.078564    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:27:55.078693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:55.078630    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:27:55.078693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:55.078664    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:27:56.114180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:56.114141    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" event={"ID":"45285d6d-5238-4608-ad26-49df796027b2","Type":"ContainerStarted","Data":"31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b"}
Nov 03 03:27:56.355982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:56.355953    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:27:56.356172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:56.356128    5515 patch_prober.go:29] interesting pod/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 03:27:56.356226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:56.356170    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 03:27:56.540072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:56.539972    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:57.356038 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:57.356008    5515 patch_prober.go:29] interesting pod/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 03:27:57.356440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:57.356056    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 03:27:57.881576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:57.880599    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:27:58.355916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:58.355835    5515 patch_prober.go:29] interesting pod/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 03:27:58.355916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:58.355895    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 03:27:59.355684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:59.355644    5515 patch_prober.go:29] interesting pod/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 03:27:59.355840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:27:59.355741    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 03:28:00.355773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:00.355732    5515 patch_prober.go:29] interesting pod/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 03:28:00.356271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:00.355801    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 03:28:00.535076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:00.535040    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:01.360216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:01.360187    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:28:01.360632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:01.360460    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:28:01.361289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:01.361273    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:28:02.723947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:02.723893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:07.621817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:07.621763    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:12.530088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:12.530049    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:13.174790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:13.174755    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:13.367947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:13.367888    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:14.133793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.133753    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"]
Nov 03 03:28:14.156306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.156271    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.182575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.182512    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.187949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.187921    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.188271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.188250    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.189052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.188590    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.189314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.189294    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.189470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.189456    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfz2\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.189674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.189655    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.189856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.189843    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.190009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.189996    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.201856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.201830    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"]
Nov 03 03:28:14.247403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.247369    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4da1f5-openshift-default-dockercfg-2bzf6\""
Nov 03 03:28:14.313689 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.303514    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.377099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.377059    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfz2\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.377381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.377365    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.377615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.377540    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.380154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.380110    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.380677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.380605    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.380954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.380926    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.381085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.381068    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.381478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.381447    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.382338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.329126    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.382748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.379085    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:14.383288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.383258    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.395182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.383682    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.395182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.386244    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.395182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.392180    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.395182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.393891    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.401537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.401507    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.441494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.441455    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfz2\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.444751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.444723    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token\") pod \"e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") " pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.504751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.504722    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:14.596671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.595400    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:14.939776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.939748    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"]
Nov 03 03:28:14.940222 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:28:14.940199    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a614825_5146_44d1_8753_a72927a9e298.slice/crio-9e59a861e45efd3071a55ac8595d9107023efa7a6fc80b11c18a2d74ff3e484e WatchSource:0}: Error finding container 9e59a861e45efd3071a55ac8595d9107023efa7a6fc80b11c18a2d74ff3e484e: Status 404 returned error can't find the container with id 9e59a861e45efd3071a55ac8595d9107023efa7a6fc80b11c18a2d74ff3e484e
Nov 03 03:28:14.942883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.942850    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:28:14.942967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.942915    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:28:14.942967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:14.942961    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:28:15.332104 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:15.332009    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:15.485574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:15.468405    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" event={"ID":"1a614825-5146-44d1-8753-a72927a9e298","Type":"ContainerStarted","Data":"c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"}
Nov 03 03:28:15.485916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:15.485885    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" event={"ID":"1a614825-5146-44d1-8753-a72927a9e298","Type":"ContainerStarted","Data":"9e59a861e45efd3071a55ac8595d9107023efa7a6fc80b11c18a2d74ff3e484e"}
Nov 03 03:28:15.510691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:15.510654    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:15.605185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:15.605066    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podStartSLOduration=1.6050473539999999 podStartE2EDuration="1.605047354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:15.601851226 +0000 UTC m=+1586.832743643" watchObservedRunningTime="2026-11-03 03:28:15.605047354 +0000 UTC m=+1586.835939775"
Nov 03 03:28:16.510383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:16.510352    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:17.489199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:17.489160    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:17.490286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:17.490263    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:28:18.238496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:18.238460    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:18.442453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:18.442423    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:18.617003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:18.616972    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:18.767693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:18.767664    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:20.984931 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:28:20.984891    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:28:21.678838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:21.678803    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:21.920938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:21.920905    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:22.105772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.102151    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"]
Nov 03 03:28:22.138899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.138871    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.241577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.217175    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"]
Nov 03 03:28:22.242055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.242031    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4da1f5\""
Nov 03 03:28:22.242394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.242376    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4da1f5-dockercfg-kzg4x\""
Nov 03 03:28:22.298582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.274682    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.298582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.274819    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjkb\" (UniqueName: \"kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb\") pod \"payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.330572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.328389    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"]
Nov 03 03:28:22.364642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.345833    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.416899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.376713    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.469900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.469864    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.470203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.470184    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjkb\" (UniqueName: \"kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb\") pod \"payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.470409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.470395    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87j72\" (UniqueName: \"kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72\") pod \"payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.476571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.471111    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.476571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.476180    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"]
Nov 03 03:28:22.532862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.532829    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjkb\" (UniqueName: \"kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb\") pod \"payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:22.571365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.571327    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.571618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.571484    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87j72\" (UniqueName: \"kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72\") pod \"payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.572116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.572093    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.634303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.634230    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87j72\" (UniqueName: \"kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72\") pod \"payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.812998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.812955    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:22.815045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:22.815013    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:23.197897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.197820    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"]
Nov 03 03:28:23.231868 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:28:23.231834    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1baaede1_f5ca_427b_a294_9718bf6481d4.slice/crio-72130971a7fc16ac8b61a80e07f1c5e68abbef7c18b374ca3968235b277db281 WatchSource:0}: Error finding container 72130971a7fc16ac8b61a80e07f1c5e68abbef7c18b374ca3968235b277db281: Status 404 returned error can't find the container with id 72130971a7fc16ac8b61a80e07f1c5e68abbef7c18b374ca3968235b277db281
Nov 03 03:28:23.248098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.248072    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"]
Nov 03 03:28:23.248636 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:28:23.248605    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61737ad_a4a3_4d6e_b163_99e6cbda21c7.slice/crio-578fc949fe1363ad35b057423f9d0169bcc2cd2d3c376dcccee79b326072816f WatchSource:0}: Error finding container 578fc949fe1363ad35b057423f9d0169bcc2cd2d3c376dcccee79b326072816f: Status 404 returned error can't find the container with id 578fc949fe1363ad35b057423f9d0169bcc2cd2d3c376dcccee79b326072816f
Nov 03 03:28:23.434608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.434577    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:23.551264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.548724    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f" event={"ID":"1baaede1-f5ca-427b-a294-9718bf6481d4","Type":"ContainerStarted","Data":"766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1"}
Nov 03 03:28:23.551264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.548801    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f" event={"ID":"1baaede1-f5ca-427b-a294-9718bf6481d4","Type":"ContainerStarted","Data":"72130971a7fc16ac8b61a80e07f1c5e68abbef7c18b374ca3968235b277db281"}
Nov 03 03:28:23.551264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.550011    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:23.553581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.553502    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw" event={"ID":"f61737ad-a4a3-4d6e-b163-99e6cbda21c7","Type":"ContainerStarted","Data":"c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953"}
Nov 03 03:28:23.553581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.553577    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw" event={"ID":"f61737ad-a4a3-4d6e-b163-99e6cbda21c7","Type":"ContainerStarted","Data":"578fc949fe1363ad35b057423f9d0169bcc2cd2d3c376dcccee79b326072816f"}
Nov 03 03:28:23.557576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.554444    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:23.670587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.670511    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f" podStartSLOduration=1.6704926009999999 podStartE2EDuration="1.670492601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:23.666719547 +0000 UTC m=+1594.897611965" watchObservedRunningTime="2026-11-03 03:28:23.670492601 +0000 UTC m=+1594.901385023"
Nov 03 03:28:23.847632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:23.846501    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw" podStartSLOduration=1.846477652 podStartE2EDuration="1.846477652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:23.828770051 +0000 UTC m=+1595.059662470" watchObservedRunningTime="2026-11-03 03:28:23.846477652 +0000 UTC m=+1595.077370072"
Nov 03 03:28:28.329695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:28.329651    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:33.231397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:33.231365    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:35.570776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:35.570745    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:28:35.571171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:35.570817    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:28:37.988493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:37.988450    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:28:38.130737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:38.130696    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:43.042223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:43.042190    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:48.130199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:48.130163    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:48.282968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:48.282930    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:48.869173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:48.869140    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:49.359301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.359272    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"]
Nov 03 03:28:49.364616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.364589    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.434926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.434784    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-398727-openshift-default-dockercfg-ckr6k\""
Nov 03 03:28:49.470201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.470168    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:49.536462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.533709    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.567778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.567734    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.568208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.568183    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.576700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.576665    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.612722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.612639    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.612994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.612972    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.639007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.613112    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tlr\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.639418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.639385    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.654575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.639690    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.671474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.599903    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"]
Nov 03 03:28:49.760997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.760966    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761022    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761060    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761078    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761100    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761137    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761189    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tlr\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761633    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761695    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761791    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761822    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.761896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.761830    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.762032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.762016    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.762079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.762017    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.764218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.764194    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.764384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.764361    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.823574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.810830    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:49.830760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.830729    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.832303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.832054    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tlr\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-742fn\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:49.990290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:49.990219    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:50.420930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.420901    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"]
Nov 03 03:28:50.421569 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:28:50.421522    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036c8e7a_11e5_4fb6_9471_23698540e2ca.slice/crio-5e2dfe90e6182b1223f08714c7ec2840d96ac6f2c07a3c1954b7eef34664da1b WatchSource:0}: Error finding container 5e2dfe90e6182b1223f08714c7ec2840d96ac6f2c07a3c1954b7eef34664da1b: Status 404 returned error can't find the container with id 5e2dfe90e6182b1223f08714c7ec2840d96ac6f2c07a3c1954b7eef34664da1b
Nov 03 03:28:50.424041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.424010    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:28:50.424121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.424084    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:28:50.424121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.424113    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:28:50.733914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.733823    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" event={"ID":"036c8e7a-11e5-4fb6-9471-23698540e2ca","Type":"ContainerStarted","Data":"ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"}
Nov 03 03:28:50.733914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.733897    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" event={"ID":"036c8e7a-11e5-4fb6-9471-23698540e2ca","Type":"ContainerStarted","Data":"5e2dfe90e6182b1223f08714c7ec2840d96ac6f2c07a3c1954b7eef34664da1b"}
Nov 03 03:28:50.857014 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:50.856965    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podStartSLOduration=1.85695121 podStartE2EDuration="1.85695121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:50.856128792 +0000 UTC m=+1622.087021209" watchObservedRunningTime="2026-11-03 03:28:50.85695121 +0000 UTC m=+1622.087843633"
Nov 03 03:28:51.013628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:51.004029    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:51.046577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:51.044277    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:51.091596 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:28:51.090847    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:28:51.740068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:51.740033    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:51.741014 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:51.740994    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:28:53.321810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:53.321722    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:53.817660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:53.817625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:53.976887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:53.976855    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:54.155870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:54.155826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:56.841963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:56.841913    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:57.026995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.026958    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:57.256574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.247142    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:57.886971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.886937    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"]
Nov 03 03:28:57.901899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.901861    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"]
Nov 03 03:28:57.904697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.902052    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:57.912030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.911805    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l85\" (UniqueName: \"kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85\") pod \"payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:57.912030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:57.911901    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:58.008729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.008692    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-398727\""
Nov 03 03:28:58.008906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.008866    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-398727-dockercfg-kkjj9\""
Nov 03 03:28:58.013695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.013673    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l85\" (UniqueName: \"kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85\") pod \"payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:58.013792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.013780    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:58.014469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.014446    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:58.076803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.075711    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l85\" (UniqueName: \"kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85\") pod \"payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:58.234719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.234621    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:58.545640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.545495    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:28:58.655572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.652771    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"]
Nov 03 03:28:58.677570 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:28:58.669639    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6527b2f_0450_4b50_83ed_19262fd9671f.slice/crio-c81f7f6a2ae99c60aaf4822d41aff514665dce5025f5027d2d3f99081d895023 WatchSource:0}: Error finding container c81f7f6a2ae99c60aaf4822d41aff514665dce5025f5027d2d3f99081d895023: Status 404 returned error can't find the container with id c81f7f6a2ae99c60aaf4822d41aff514665dce5025f5027d2d3f99081d895023
Nov 03 03:28:58.906898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.906866    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm" event={"ID":"b6527b2f-0450-4b50-83ed-19262fd9671f","Type":"ContainerStarted","Data":"9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246"}
Nov 03 03:28:58.907262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.906910    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm" event={"ID":"b6527b2f-0450-4b50-83ed-19262fd9671f","Type":"ContainerStarted","Data":"c81f7f6a2ae99c60aaf4822d41aff514665dce5025f5027d2d3f99081d895023"}
Nov 03 03:28:58.907262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:58.907037    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:28:59.025853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:28:59.025799    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm" podStartSLOduration=2.025783963 podStartE2EDuration="2.025783963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:28:59.023979368 +0000 UTC m=+1630.254871787" watchObservedRunningTime="2026-11-03 03:28:59.025783963 +0000 UTC m=+1630.256676381"
Nov 03 03:29:03.287456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:03.287412    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:03.459898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:03.455625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:03.856503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:03.856467    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:08.298672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:08.298638    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:09.106558 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:09.106525    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:09.916678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:09.916650    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:29:10.273751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:10.273656    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:29:13.269172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:13.269137    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:14.897467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:14.897438    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:15.437439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.437409    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"]
Nov 03 03:29:15.441822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.441805    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.498777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.498750    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-64c59d-openshift-default-dockercfg-vgv2p\""
Nov 03 03:29:15.505231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.505197    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"]
Nov 03 03:29:15.527910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.527883    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:15.553131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553103    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4q2r\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553182    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553275    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553321    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553479    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553516    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553640    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553670    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.553788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.553774    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.663169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.663135    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.668385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.668015    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.684195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.684113    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4q2r\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.684672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.684636    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.686254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.685392    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.686438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.686220    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.686987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.686971    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.687139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.686943    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.703616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.701473    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.704003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.703979    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.704582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.704474    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.704821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.704794    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.705964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.705918    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.706564 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.706517    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.736835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.736216    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.758020 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.757035    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.778580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.768732    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4q2r\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.802117 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.802085    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token\") pod \"e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") " pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:15.948386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:15.948323    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:16.057418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:16.057341    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:16.491357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:16.491329    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"]
Nov 03 03:29:16.492445 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:29:16.492418    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bbf043_42ca_4359_92b7_fb8ec834da65.slice/crio-9d82514614369f468e1def3c2e8c67ab8cbe5f5430bb6eeb69204e7c872557e2 WatchSource:0}: Error finding container 9d82514614369f468e1def3c2e8c67ab8cbe5f5430bb6eeb69204e7c872557e2: Status 404 returned error can't find the container with id 9d82514614369f468e1def3c2e8c67ab8cbe5f5430bb6eeb69204e7c872557e2
Nov 03 03:29:16.495164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:16.495132    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:29:16.495214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:16.495201    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:29:16.495254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:16.495231    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:29:17.042576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:17.039735    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" event={"ID":"a4bbf043-42ca-4359-92b7-fb8ec834da65","Type":"ContainerStarted","Data":"28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d"}
Nov 03 03:29:17.042576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:17.039807    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" event={"ID":"a4bbf043-42ca-4359-92b7-fb8ec834da65","Type":"ContainerStarted","Data":"9d82514614369f468e1def3c2e8c67ab8cbe5f5430bb6eeb69204e7c872557e2"}
Nov 03 03:29:17.057530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:17.057497    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:17.063335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:17.063313    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:17.173152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:17.173103    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podStartSLOduration=2.173089281 podStartE2EDuration="2.173089281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:17.171861615 +0000 UTC m=+1648.402754033" watchObservedRunningTime="2026-11-03 03:29:17.173089281 +0000 UTC m=+1648.403981762"
Nov 03 03:29:18.046716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:18.046687    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:18.047838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:18.047816    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:29:18.235021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:18.234988    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:20.002035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:20.001995    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:20.170111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:20.170079    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:20.315480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:20.315404    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:20.984800 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:29:20.984760    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:29:23.216879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.216847    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:23.422293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.422242    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:23.624021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.623957    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:23.805887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.805731    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"]
Nov 03 03:29:23.816572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.816525    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:23.877864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.877790    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"]
Nov 03 03:29:23.937580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.923000    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-64c59d-dockercfg-mx8xk\""
Nov 03 03:29:23.937580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:23.923081    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-64c59d\""
Nov 03 03:29:24.020471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.007877    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftd8\" (UniqueName: \"kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8\") pod \"payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") " pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.021018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.020986    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") " pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.136607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.136501    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftd8\" (UniqueName: \"kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8\") pod \"payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") " pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.137167 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.136631    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") " pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.137409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.137388    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") " pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.198745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.198709    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftd8\" (UniqueName: \"kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8\") pod \"payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") " pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.332470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.332436    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:24.709313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.709285    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"]
Nov 03 03:29:24.710039 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:29:24.710014    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a5132a_039c_480e_a4c0_9a36427eb20c.slice/crio-5b12f62278c8974996c096551ed8a1840bdc4e44ae098acd86d06fc74ca07611 WatchSource:0}: Error finding container 5b12f62278c8974996c096551ed8a1840bdc4e44ae098acd86d06fc74ca07611: Status 404 returned error can't find the container with id 5b12f62278c8974996c096551ed8a1840bdc4e44ae098acd86d06fc74ca07611
Nov 03 03:29:24.901209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:24.901181    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:25.107504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:25.107479    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt" event={"ID":"f0a5132a-039c-480e-a4c0-9a36427eb20c","Type":"ContainerStarted","Data":"727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399"}
Nov 03 03:29:25.107676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:25.107531    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt" event={"ID":"f0a5132a-039c-480e-a4c0-9a36427eb20c","Type":"ContainerStarted","Data":"5b12f62278c8974996c096551ed8a1840bdc4e44ae098acd86d06fc74ca07611"}
Nov 03 03:29:25.107676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:25.107575    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:25.227412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:25.227361    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt" podStartSLOduration=2.227345904 podStartE2EDuration="2.227345904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:25.22512848 +0000 UTC m=+1656.456020911" watchObservedRunningTime="2026-11-03 03:29:25.227345904 +0000 UTC m=+1656.458238322"
Nov 03 03:29:25.498842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:25.498750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:25.817690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:25.811778    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:28.752467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:28.752430    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:30.202266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:30.202227    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:30.449635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:30.449597    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:30.600538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:30.600457    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:33.735293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:33.735263    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:34.135587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.135273    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:34.176665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.176628    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"]
Nov 03 03:29:34.182491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.182464    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.239430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.239394    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"]
Nov 03 03:29:34.555190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.487083    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-372186-dockercfg-w8rpp\""
Nov 03 03:29:34.641710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.491401    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-372186\""
Nov 03 03:29:34.641948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.538842    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:34.641948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.541273    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwltx\" (UniqueName: \"kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx\") pod \"payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.642215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.642193    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.642399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.568377    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"]
Nov 03 03:29:34.650301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.650270    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"]
Nov 03 03:29:34.650658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.650381    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.743180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.743147    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.743644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.743221    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4982\" (UniqueName: \"kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982\") pod \"payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") " pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.743644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.743329    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwltx\" (UniqueName: \"kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx\") pod \"payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.743644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.743577    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume\") pod \"payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") " pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.743958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.743941    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.805633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.805536    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwltx\" (UniqueName: \"kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx\") pod \"payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:34.845542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.845503    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume\") pod \"payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") " pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.845730 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.845629    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4982\" (UniqueName: \"kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982\") pod \"payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") " pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.846244 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.846224    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume\") pod \"payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") " pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.908240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.908198    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4982\" (UniqueName: \"kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982\") pod \"payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") " pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:34.997121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:34.997082    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:35.100664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:35.099355    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:35.389254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:35.389222    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"]
Nov 03 03:29:35.392111 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:29:35.392083    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0f56d0_f2da_4506_bda0_9b971be15163.slice/crio-9d120c0ed45c15877e52a616d4a95570f9d930d14a12cb7270a13af03a485f8d WatchSource:0}: Error finding container 9d120c0ed45c15877e52a616d4a95570f9d930d14a12cb7270a13af03a485f8d: Status 404 returned error can't find the container with id 9d120c0ed45c15877e52a616d4a95570f9d930d14a12cb7270a13af03a485f8d
Nov 03 03:29:35.526813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:35.526787    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"]
Nov 03 03:29:35.527391 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:29:35.527355    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfaf6c4_1871_464a_8839_76c4100c7e79.slice/crio-1cca9820282c05583abaaf53161609628eb664d7e7770d81c39d9d7fef1adf2b WatchSource:0}: Error finding container 1cca9820282c05583abaaf53161609628eb664d7e7770d81c39d9d7fef1adf2b: Status 404 returned error can't find the container with id 1cca9820282c05583abaaf53161609628eb664d7e7770d81c39d9d7fef1adf2b
Nov 03 03:29:36.117349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.117321    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:29:36.221744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.221707    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t" event={"ID":"4d0f56d0-f2da-4506-bda0-9b971be15163","Type":"ContainerStarted","Data":"0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb"}
Nov 03 03:29:36.221942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.221778    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:36.221942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.221806    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t" event={"ID":"4d0f56d0-f2da-4506-bda0-9b971be15163","Type":"ContainerStarted","Data":"9d120c0ed45c15877e52a616d4a95570f9d930d14a12cb7270a13af03a485f8d"}
Nov 03 03:29:36.223362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.223332    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h" event={"ID":"cbfaf6c4-1871-464a-8839-76c4100c7e79","Type":"ContainerStarted","Data":"dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f"}
Nov 03 03:29:36.223479 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.223380    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h" event={"ID":"cbfaf6c4-1871-464a-8839-76c4100c7e79","Type":"ContainerStarted","Data":"1cca9820282c05583abaaf53161609628eb664d7e7770d81c39d9d7fef1adf2b"}
Nov 03 03:29:36.223479 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.223467    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:36.393647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.391055    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h" podStartSLOduration=2.391033394 podStartE2EDuration="2.391033394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:36.38773488 +0000 UTC m=+1667.618627301" watchObservedRunningTime="2026-11-03 03:29:36.391033394 +0000 UTC m=+1667.621925813"
Nov 03 03:29:36.543571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:36.542076    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t" podStartSLOduration=2.542052647 podStartE2EDuration="2.542052647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:29:36.515016389 +0000 UTC m=+1667.745908808" watchObservedRunningTime="2026-11-03 03:29:36.542052647 +0000 UTC m=+1667.772945068"
Nov 03 03:29:37.717332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:37.717297    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:37.991693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:37.953499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:38.806136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:38.806101    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:43.326764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.326732    5515 generic.go:356] "Generic (PLEG): container finished" podID="1a614825-5146-44d1-8753-a72927a9e298" containerID="c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7" exitCode=0
Nov 03 03:29:43.327208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.326807    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" event={"ID":"1a614825-5146-44d1-8753-a72927a9e298","Type":"ContainerDied","Data":"c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"}
Nov 03 03:29:43.328525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.327937    5515 scope.go:122] "RemoveContainer" containerID="c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"
Nov 03 03:29:43.337911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.337870    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:29:43.338043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.337946    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:29:43.338043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.337991    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:29:43.579425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.579332    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:43.773575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.770109    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:43.916499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:43.916463    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:44.336108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:44.336074    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" event={"ID":"1a614825-5146-44d1-8753-a72927a9e298","Type":"ContainerStarted","Data":"3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70"}
Nov 03 03:29:44.505757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:44.505719    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:29:44.507246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:44.507221    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 03:29:44.507365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:44.507283    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 03:29:45.505485 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:45.505442    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 03:29:45.506022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:45.505506    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 03:29:46.509461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:46.509424    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 03:29:46.510265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:46.510229    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 03:29:47.233760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:47.233730    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:29:47.233974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:47.233884    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:29:47.495195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:47.495103    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:29:47.505611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:47.505343    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 03:29:47.505611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:47.505402    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 03:29:48.505799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:48.505758    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 03:29:48.506207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:48.505819    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 03:29:48.632230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:48.632195    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:49.505615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:49.505538    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 03:29:49.505802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:49.505632    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 03:29:50.510620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:50.510582    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:29:50.511789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:50.511766    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:29:50.986480 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:29:50.986445    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:29:53.502976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:53.502943    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:57.194977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:57.194938    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:29:58.322842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:29:58.322801    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:00.177128 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.177092    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"]
Nov 03 03:30:00.182925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.182901    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"]
Nov 03 03:30:00.183054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.183033    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.187246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.187225    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.199785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.199765    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"]
Nov 03 03:30:00.203839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.203824    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.228126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.228100    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"]
Nov 03 03:30:00.232301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.232284    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"]
Nov 03 03:30:00.232468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.232450    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.236928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.236897    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.246886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.246866    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\""
Nov 03 03:30:00.281895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.281861    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"]
Nov 03 03:30:00.285304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.285278    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"]
Nov 03 03:30:00.347875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.347694    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2z6s\" (UniqueName: \"kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s\") pod \"maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.347875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.347791    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.347875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.347826    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.347897    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwmh\" (UniqueName: \"kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh\") pod \"maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.347965    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.348004    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxpp\" (UniqueName: \"kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.348075    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjpn\" (UniqueName: \"kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.348109    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9m5x\" (UniqueName: \"kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x\") pod \"maas-api-key-cleanup-29894610-cb898\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.348152    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894610-cb898\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.348306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.348275    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.350656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.350628    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"]
Nov 03 03:30:00.447850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.432572    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"]
Nov 03 03:30:00.582337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.539751    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"]
Nov 03 03:30:00.587449 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.584568    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.588810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.588784    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxpp\" (UniqueName: \"kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.589075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.589026    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjpn\" (UniqueName: \"kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.589397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.589378    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9m5x\" (UniqueName: \"kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x\") pod \"maas-api-key-cleanup-29894610-cb898\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.589600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.589582    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894610-cb898\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.589848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.589831    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.590073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.590057    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2z6s\" (UniqueName: \"kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s\") pod \"maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.590233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.590218    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.590339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.590326    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.590697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.590678    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwmh\" (UniqueName: \"kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh\") pod \"maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.591918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.585386    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.606271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.606237    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:30:00.634881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.634842    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894610-cb898\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.635820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.635783    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.636669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.636640    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.637491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.637465    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.655255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.655216    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9m5x\" (UniqueName: \"kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x\") pod \"maas-api-key-cleanup-29894610-cb898\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:00.697364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.697330    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwmh\" (UniqueName: \"kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh\") pod \"maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.697523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.697420    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxpp\" (UniqueName: \"kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.697523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.697420    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjpn\" (UniqueName: \"kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.698008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.697958    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2z6s\" (UniqueName: \"kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s\") pod \"maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.799507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.799478    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:00.805847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.805825    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:00.820822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.820805    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:00.861001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.852090    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:00.895619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:00.875225    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:01.335196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.334670    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"]
Nov 03 03:30:01.486442 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:30:01.486413    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce378bb_7357_46ae_8159_065d109e2baa.slice/crio-067305423f40c376c06719d984640722f8c461c2bf8bebddb38dcde4f2d0755b WatchSource:0}: Error finding container 067305423f40c376c06719d984640722f8c461c2bf8bebddb38dcde4f2d0755b: Status 404 returned error can't find the container with id 067305423f40c376c06719d984640722f8c461c2bf8bebddb38dcde4f2d0755b
Nov 03 03:30:01.486580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.486488    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"]
Nov 03 03:30:01.535525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.535408    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"]
Nov 03 03:30:01.535754 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:30:01.535721    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb39043a_38b8_421b_825b_240ffbd44ec4.slice/crio-dd98905ddf4600c1a9db6c3b62f981fbbec46c70e75cce3112ad7db6da055621 WatchSource:0}: Error finding container dd98905ddf4600c1a9db6c3b62f981fbbec46c70e75cce3112ad7db6da055621: Status 404 returned error can't find the container with id dd98905ddf4600c1a9db6c3b62f981fbbec46c70e75cce3112ad7db6da055621
Nov 03 03:30:01.538150 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:30:01.538121    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4df851_36f7_4c6d_8cb2_a4a6d89de81a.slice/crio-fff22ed0338f982f3a1087dcd22530356d36e45e1ab29094b5e1a7d495346275 WatchSource:0}: Error finding container fff22ed0338f982f3a1087dcd22530356d36e45e1ab29094b5e1a7d495346275: Status 404 returned error can't find the container with id fff22ed0338f982f3a1087dcd22530356d36e45e1ab29094b5e1a7d495346275
Nov 03 03:30:01.539689 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.539662    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"]
Nov 03 03:30:01.564003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.563926    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"]
Nov 03 03:30:01.574500 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:30:01.574475    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc36b093_09b4_43f3_88c4_294c70a0dbb5.slice/crio-6363c39b95f22b3f438d1cac680b39a2ffd637ebf671e96f05b27b792f851b5c WatchSource:0}: Error finding container 6363c39b95f22b3f438d1cac680b39a2ffd637ebf671e96f05b27b792f851b5c: Status 404 returned error can't find the container with id 6363c39b95f22b3f438d1cac680b39a2ffd637ebf671e96f05b27b792f851b5c
Nov 03 03:30:01.646726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.646689    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" event={"ID":"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a","Type":"ContainerStarted","Data":"fff22ed0338f982f3a1087dcd22530356d36e45e1ab29094b5e1a7d495346275"}
Nov 03 03:30:01.648047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.648017    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" event={"ID":"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a","Type":"ContainerStarted","Data":"cd21c921c928b6983d60ad8659864d2d6b6de4d3643e9751ebb193978365b2d9"}
Nov 03 03:30:01.649251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.649232    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" event={"ID":"bc36b093-09b4-43f3-88c4-294c70a0dbb5","Type":"ContainerStarted","Data":"6363c39b95f22b3f438d1cac680b39a2ffd637ebf671e96f05b27b792f851b5c"}
Nov 03 03:30:01.650511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.650491    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerStarted","Data":"dd98905ddf4600c1a9db6c3b62f981fbbec46c70e75cce3112ad7db6da055621"}
Nov 03 03:30:01.652026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:01.651994    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerStarted","Data":"067305423f40c376c06719d984640722f8c461c2bf8bebddb38dcde4f2d0755b"}
Nov 03 03:30:02.784574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.783470    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerStarted","Data":"8d316bf36558a0b8548623c0a011e0d160bae301901e1875eecf19536659c37f"}
Nov 03 03:30:02.787740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.787701    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" event={"ID":"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a","Type":"ContainerStarted","Data":"53f66e2553fbc8428db591b34ebc97b5ba44b0c8e4d6211561b888d1b771d3a8"}
Nov 03 03:30:02.791971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.791942    5515 generic.go:356] "Generic (PLEG): container finished" podID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerID="a1ff4d8e47f1b0f9b77b262aa0170c0c17817f3f22dd035f9ea794ec8e2ec109" exitCode=60
Nov 03 03:30:02.792169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.792018    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" event={"ID":"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a","Type":"ContainerDied","Data":"a1ff4d8e47f1b0f9b77b262aa0170c0c17817f3f22dd035f9ea794ec8e2ec109"}
Nov 03 03:30:02.792446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.792265    5515 scope.go:122] "RemoveContainer" containerID="a1ff4d8e47f1b0f9b77b262aa0170c0c17817f3f22dd035f9ea794ec8e2ec109"
Nov 03 03:30:02.793918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.793895    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" event={"ID":"bc36b093-09b4-43f3-88c4-294c70a0dbb5","Type":"ContainerStarted","Data":"151f623200e7a9cc05b3d925acb12514dd8554c7d455a166386a023ce7bb7612"}
Nov 03 03:30:02.796213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.796116    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerStarted","Data":"e8dffd821a6914863f73f69326d1b9233fcd79bcd6b18a9e32dd7aabaf009809"}
Nov 03 03:30:02.796371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.796356    5515 scope.go:122] "RemoveContainer" containerID="e8dffd821a6914863f73f69326d1b9233fcd79bcd6b18a9e32dd7aabaf009809"
Nov 03 03:30:02.906821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:02.906689    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" podStartSLOduration=2.906668656 podStartE2EDuration="2.906668656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:02.90564789 +0000 UTC m=+1694.136540306" watchObservedRunningTime="2026-11-03 03:30:02.906668656 +0000 UTC m=+1694.137561078"
Nov 03 03:30:03.254794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.254753    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:03.264005 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.263957    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" podStartSLOduration=3.263943395 podStartE2EDuration="3.263943395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:03.260210283 +0000 UTC m=+1694.491102702" watchObservedRunningTime="2026-11-03 03:30:03.263943395 +0000 UTC m=+1694.494835813"
Nov 03 03:30:03.410043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.409976    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" podStartSLOduration=3.409955985 podStartE2EDuration="3.409955985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:03.404128845 +0000 UTC m=+1694.635021264" watchObservedRunningTime="2026-11-03 03:30:03.409955985 +0000 UTC m=+1694.640848405"
Nov 03 03:30:03.805634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.805600    5515 generic.go:356] "Generic (PLEG): container finished" podID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerID="53f66e2553fbc8428db591b34ebc97b5ba44b0c8e4d6211561b888d1b771d3a8" exitCode=60
Nov 03 03:30:03.806036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.805700    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" event={"ID":"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a","Type":"ContainerDied","Data":"53f66e2553fbc8428db591b34ebc97b5ba44b0c8e4d6211561b888d1b771d3a8"}
Nov 03 03:30:03.806036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.805964    5515 scope.go:122] "RemoveContainer" containerID="53f66e2553fbc8428db591b34ebc97b5ba44b0c8e4d6211561b888d1b771d3a8"
Nov 03 03:30:03.807801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.807781    5515 generic.go:356] "Generic (PLEG): container finished" podID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerID="ba3eaec52bcf169d0feb813d0af18c7ec234ffb6e671cd9f728ea625b4d8c466" exitCode=60
Nov 03 03:30:03.807901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.807814    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" event={"ID":"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a","Type":"ContainerDied","Data":"ba3eaec52bcf169d0feb813d0af18c7ec234ffb6e671cd9f728ea625b4d8c466"}
Nov 03 03:30:03.807901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.807885    5515 scope.go:122] "RemoveContainer" containerID="a1ff4d8e47f1b0f9b77b262aa0170c0c17817f3f22dd035f9ea794ec8e2ec109"
Nov 03 03:30:03.808143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.808124    5515 scope.go:122] "RemoveContainer" containerID="ba3eaec52bcf169d0feb813d0af18c7ec234ffb6e671cd9f728ea625b4d8c466"
Nov 03 03:30:03.808475 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:03.808452    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4_odh-ai-gateway-infra(9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a"
Nov 03 03:30:03.809404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.809385    5515 generic.go:356] "Generic (PLEG): container finished" podID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerID="151f623200e7a9cc05b3d925acb12514dd8554c7d455a166386a023ce7bb7612" exitCode=60
Nov 03 03:30:03.809516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.809467    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" event={"ID":"bc36b093-09b4-43f3-88c4-294c70a0dbb5","Type":"ContainerDied","Data":"151f623200e7a9cc05b3d925acb12514dd8554c7d455a166386a023ce7bb7612"}
Nov 03 03:30:03.809864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.809840    5515 scope.go:122] "RemoveContainer" containerID="151f623200e7a9cc05b3d925acb12514dd8554c7d455a166386a023ce7bb7612"
Nov 03 03:30:03.811586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.811562    5515 generic.go:356] "Generic (PLEG): container finished" podID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerID="e8dffd821a6914863f73f69326d1b9233fcd79bcd6b18a9e32dd7aabaf009809" exitCode=60
Nov 03 03:30:03.811669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.811654    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerDied","Data":"e8dffd821a6914863f73f69326d1b9233fcd79bcd6b18a9e32dd7aabaf009809"}
Nov 03 03:30:03.811719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.811696    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerStarted","Data":"7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8"}
Nov 03 03:30:03.813523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.813503    5515 generic.go:356] "Generic (PLEG): container finished" podID="8ce378bb-7357-46ae-8159-065d109e2baa" containerID="8d316bf36558a0b8548623c0a011e0d160bae301901e1875eecf19536659c37f" exitCode=60
Nov 03 03:30:03.813664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.813534    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerDied","Data":"8d316bf36558a0b8548623c0a011e0d160bae301901e1875eecf19536659c37f"}
Nov 03 03:30:03.813914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:03.813898    5515 scope.go:122] "RemoveContainer" containerID="8d316bf36558a0b8548623c0a011e0d160bae301901e1875eecf19536659c37f"
Nov 03 03:30:04.169337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.169290    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" podStartSLOduration=4.169275145 podStartE2EDuration="4.169275145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:04.050990968 +0000 UTC m=+1695.281883383" watchObservedRunningTime="2026-11-03 03:30:04.169275145 +0000 UTC m=+1695.400167562"
Nov 03 03:30:04.824568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.824515    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerStarted","Data":"3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c"}
Nov 03 03:30:04.826427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.826404    5515 generic.go:356] "Generic (PLEG): container finished" podID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerID="14059d10231567cd0b254b7a59dd743487eb77308e135255472b2ca11585f6c3" exitCode=60
Nov 03 03:30:04.826584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.826484    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" event={"ID":"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a","Type":"ContainerDied","Data":"14059d10231567cd0b254b7a59dd743487eb77308e135255472b2ca11585f6c3"}
Nov 03 03:30:04.826584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.826542    5515 scope.go:122] "RemoveContainer" containerID="53f66e2553fbc8428db591b34ebc97b5ba44b0c8e4d6211561b888d1b771d3a8"
Nov 03 03:30:04.826767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.826751    5515 scope.go:122] "RemoveContainer" containerID="14059d10231567cd0b254b7a59dd743487eb77308e135255472b2ca11585f6c3"
Nov 03 03:30:04.827088 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:04.827065    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k_odh-ai-gateway-infra(fd4df851-36f7-4c6d-8cb2-a4a6d89de81a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a"
Nov 03 03:30:04.828351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.828335    5515 scope.go:122] "RemoveContainer" containerID="ba3eaec52bcf169d0feb813d0af18c7ec234ffb6e671cd9f728ea625b4d8c466"
Nov 03 03:30:04.828686 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:04.828663    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4_odh-ai-gateway-infra(9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a"
Nov 03 03:30:04.830054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.830031    5515 generic.go:356] "Generic (PLEG): container finished" podID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerID="07192e919c62f3da09c9c2111a9f7a36e178bb05bfcd8933e662260f4ec84b73" exitCode=60
Nov 03 03:30:04.830157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.830103    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" event={"ID":"bc36b093-09b4-43f3-88c4-294c70a0dbb5","Type":"ContainerDied","Data":"07192e919c62f3da09c9c2111a9f7a36e178bb05bfcd8933e662260f4ec84b73"}
Nov 03 03:30:04.830332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.830319    5515 scope.go:122] "RemoveContainer" containerID="07192e919c62f3da09c9c2111a9f7a36e178bb05bfcd8933e662260f4ec84b73"
Nov 03 03:30:04.830636 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:04.830617    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894610-cb898_odh-ai-gateway-infra(bc36b093-09b4-43f3-88c4-294c70a0dbb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5"
Nov 03 03:30:04.831990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.831972    5515 generic.go:356] "Generic (PLEG): container finished" podID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerID="7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8" exitCode=60
Nov 03 03:30:04.832067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.832046    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerDied","Data":"7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8"}
Nov 03 03:30:04.832222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.832209    5515 scope.go:122] "RemoveContainer" containerID="7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8"
Nov 03 03:30:04.832436 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:04.832420    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q_odh-ai-gateway-infra(eb39043a-38b8-421b-825b-240ffbd44ec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4"
Nov 03 03:30:04.840662 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.840640    5515 scope.go:122] "RemoveContainer" containerID="151f623200e7a9cc05b3d925acb12514dd8554c7d455a166386a023ce7bb7612"
Nov 03 03:30:04.854315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:04.854290    5515 scope.go:122] "RemoveContainer" containerID="e8dffd821a6914863f73f69326d1b9233fcd79bcd6b18a9e32dd7aabaf009809"
Nov 03 03:30:05.560828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.560791    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:05.845357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.845269    5515 generic.go:356] "Generic (PLEG): container finished" podID="8ce378bb-7357-46ae-8159-065d109e2baa" containerID="3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c" exitCode=60
Nov 03 03:30:05.845862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.845352    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerDied","Data":"3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c"}
Nov 03 03:30:05.845862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.845420    5515 scope.go:122] "RemoveContainer" containerID="8d316bf36558a0b8548623c0a011e0d160bae301901e1875eecf19536659c37f"
Nov 03 03:30:05.845862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.845627    5515 scope.go:122] "RemoveContainer" containerID="3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c"
Nov 03 03:30:05.846040 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:05.845986    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv_odh-ai-gateway-infra(8ce378bb-7357-46ae-8159-065d109e2baa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" podUID="8ce378bb-7357-46ae-8159-065d109e2baa"
Nov 03 03:30:05.847782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.847765    5515 scope.go:122] "RemoveContainer" containerID="14059d10231567cd0b254b7a59dd743487eb77308e135255472b2ca11585f6c3"
Nov 03 03:30:05.848056 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:05.848034    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k_odh-ai-gateway-infra(fd4df851-36f7-4c6d-8cb2-a4a6d89de81a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a"
Nov 03 03:30:05.849325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.849308    5515 scope.go:122] "RemoveContainer" containerID="07192e919c62f3da09c9c2111a9f7a36e178bb05bfcd8933e662260f4ec84b73"
Nov 03 03:30:05.849644 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:05.849625    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894610-cb898_odh-ai-gateway-infra(bc36b093-09b4-43f3-88c4-294c70a0dbb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5"
Nov 03 03:30:05.850833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:05.850820    5515 scope.go:122] "RemoveContainer" containerID="7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8"
Nov 03 03:30:05.851050 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:05.851033    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q_odh-ai-gateway-infra(eb39043a-38b8-421b-825b-240ffbd44ec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4"
Nov 03 03:30:06.859219 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:06.859190    5515 scope.go:122] "RemoveContainer" containerID="3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c"
Nov 03 03:30:06.859639 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:06.859437    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv_odh-ai-gateway-infra(8ce378bb-7357-46ae-8159-065d109e2baa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" podUID="8ce378bb-7357-46ae-8159-065d109e2baa"
Nov 03 03:30:07.719199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:07.719159    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:08.581474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:08.581422    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:08.749185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:08.749143    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:08.923445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:08.923409    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:12.844384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:12.844348    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"]
Nov 03 03:30:12.965180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:12.965145    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"]
Nov 03 03:30:13.201820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:13.201787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:14.747757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:14.747716    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:14.887490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:14.887463    5515 scope.go:122] "RemoveContainer" containerID="7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8"
Nov 03 03:30:15.986994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:15.986906    5515 generic.go:356] "Generic (PLEG): container finished" podID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerID="dd5dd13994c2d0d875776c8c851505170502b014d36ab63262d278d1b19aa848" exitCode=60
Nov 03 03:30:15.986994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:15.986982    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerDied","Data":"dd5dd13994c2d0d875776c8c851505170502b014d36ab63262d278d1b19aa848"}
Nov 03 03:30:15.987469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:15.987040    5515 scope.go:122] "RemoveContainer" containerID="7e308c5cf0ed72654552625cda8f29d0259a269d3a9c1939c2fe7b49303190c8"
Nov 03 03:30:15.987533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:15.987486    5515 scope.go:122] "RemoveContainer" containerID="dd5dd13994c2d0d875776c8c851505170502b014d36ab63262d278d1b19aa848"
Nov 03 03:30:15.987898 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:15.987875    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q_odh-ai-gateway-infra(eb39043a-38b8-421b-825b-240ffbd44ec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4"
Nov 03 03:30:16.886841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:16.886811    5515 scope.go:122] "RemoveContainer" containerID="07192e919c62f3da09c9c2111a9f7a36e178bb05bfcd8933e662260f4ec84b73"
Nov 03 03:30:16.887018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:16.886954    5515 scope.go:122] "RemoveContainer" containerID="ba3eaec52bcf169d0feb813d0af18c7ec234ffb6e671cd9f728ea625b4d8c466"
Nov 03 03:30:17.124579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.124524    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"]
Nov 03 03:30:17.302789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.302766    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:17.390369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.390339    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca\") pod \"eb39043a-38b8-421b-825b-240ffbd44ec4\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") "
Nov 03 03:30:17.390669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.390653    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn\" (UniqueName: \"kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn\") pod \"eb39043a-38b8-421b-825b-240ffbd44ec4\" (UID: \"eb39043a-38b8-421b-825b-240ffbd44ec4\") "
Nov 03 03:30:17.390725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.390685    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca" pod "eb39043a-38b8-421b-825b-240ffbd44ec4" (UID: "eb39043a-38b8-421b-825b-240ffbd44ec4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:17.391233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.391217    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb39043a-38b8-421b-825b-240ffbd44ec4-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:17.393006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.392988    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn" pod "eb39043a-38b8-421b-825b-240ffbd44ec4" (UID: "eb39043a-38b8-421b-825b-240ffbd44ec4"). InnerVolumeSpecName "kube-api-access-fhjpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:17.492530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.492442    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjpn\" (UniqueName: \"kubernetes.io/projected/eb39043a-38b8-421b-825b-240ffbd44ec4-kube-api-access-fhjpn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:17.845324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.845281    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf" podUID="7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" containerName="payload-processing" containerID="cri-o://3f2e491591694884536c3b5201903fbdf7fd91453cdf7a728b9e3cb7049565c2" gracePeriod=25
Nov 03 03:30:17.887391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.887364    5515 scope.go:122] "RemoveContainer" containerID="3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c"
Nov 03 03:30:17.966347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:17.966307    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x" podUID="d659c741-776c-4043-a094-2d9e5b653777" containerName="payload-pre-processing" containerID="cri-o://c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8" gracePeriod=25
Nov 03 03:30:18.006020 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.005984    5515 generic.go:356] "Generic (PLEG): container finished" podID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerID="06c70787fc86aa36dcbfa0db05f85723a0aac9bbbe85a2d52ed184124c3704ce" exitCode=60
Nov 03 03:30:18.006200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.006052    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" event={"ID":"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a","Type":"ContainerDied","Data":"06c70787fc86aa36dcbfa0db05f85723a0aac9bbbe85a2d52ed184124c3704ce"}
Nov 03 03:30:18.006200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.006122    5515 scope.go:122] "RemoveContainer" containerID="ba3eaec52bcf169d0feb813d0af18c7ec234ffb6e671cd9f728ea625b4d8c466"
Nov 03 03:30:18.006635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.006601    5515 scope.go:122] "RemoveContainer" containerID="06c70787fc86aa36dcbfa0db05f85723a0aac9bbbe85a2d52ed184124c3704ce"
Nov 03 03:30:18.006971 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:18.006945    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4_odh-ai-gateway-infra(9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a"
Nov 03 03:30:18.007905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.007884    5515 generic.go:356] "Generic (PLEG): container finished" podID="7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" containerID="3f2e491591694884536c3b5201903fbdf7fd91453cdf7a728b9e3cb7049565c2" exitCode=0
Nov 03 03:30:18.008003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.007955    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf" event={"ID":"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18","Type":"ContainerDied","Data":"3f2e491591694884536c3b5201903fbdf7fd91453cdf7a728b9e3cb7049565c2"}
Nov 03 03:30:18.009977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.009955    5515 generic.go:356] "Generic (PLEG): container finished" podID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerID="993a65dd7bed21d379c8393223e26fdc79ee7d2608c2ef0f41380d42ab96bb82" exitCode=60
Nov 03 03:30:18.010078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.010022    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" event={"ID":"bc36b093-09b4-43f3-88c4-294c70a0dbb5","Type":"ContainerDied","Data":"993a65dd7bed21d379c8393223e26fdc79ee7d2608c2ef0f41380d42ab96bb82"}
Nov 03 03:30:18.010445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.010429    5515 scope.go:122] "RemoveContainer" containerID="993a65dd7bed21d379c8393223e26fdc79ee7d2608c2ef0f41380d42ab96bb82"
Nov 03 03:30:18.010777 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:18.010757    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894610-cb898_odh-ai-gateway-infra(bc36b093-09b4-43f3-88c4-294c70a0dbb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5"
Nov 03 03:30:18.011446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.011424    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q" event={"ID":"eb39043a-38b8-421b-825b-240ffbd44ec4","Type":"ContainerDied","Data":"dd98905ddf4600c1a9db6c3b62f981fbbec46c70e75cce3112ad7db6da055621"}
Nov 03 03:30:18.011540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.011450    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"
Nov 03 03:30:18.085255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.085221    5515 scope.go:122] "RemoveContainer" containerID="07192e919c62f3da09c9c2111a9f7a36e178bb05bfcd8933e662260f4ec84b73"
Nov 03 03:30:18.102275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.102253    5515 scope.go:122] "RemoveContainer" containerID="dd5dd13994c2d0d875776c8c851505170502b014d36ab63262d278d1b19aa848"
Nov 03 03:30:18.207989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.207962    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:30:18.251198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.250569    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"]
Nov 03 03:30:18.256766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.256737    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894610-bnm5q"]
Nov 03 03:30:18.303700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.303674    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume\") pod \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") "
Nov 03 03:30:18.303868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.303855    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2\" (UniqueName: \"kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2\") pod \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\" (UID: \"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18\") "
Nov 03 03:30:18.304077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.304045    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume" pod "7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" (UID: "7eea000d-fcfa-43cd-98d7-acb5f5ed9b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:18.304369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.304349    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:18.306176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.306154    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2" pod "7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" (UID: "7eea000d-fcfa-43cd-98d7-acb5f5ed9b18"). InnerVolumeSpecName "kube-api-access-dfqv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:18.330301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.330277    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:30:18.405457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.405420    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx\" (UniqueName: \"kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx\") pod \"d659c741-776c-4043-a094-2d9e5b653777\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") "
Nov 03 03:30:18.405721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.405704    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume\") pod \"d659c741-776c-4043-a094-2d9e5b653777\" (UID: \"d659c741-776c-4043-a094-2d9e5b653777\") "
Nov 03 03:30:18.406082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.406059    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume" pod "d659c741-776c-4043-a094-2d9e5b653777" (UID: "d659c741-776c-4043-a094-2d9e5b653777"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:18.406348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.406334    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqv2\" (UniqueName: \"kubernetes.io/projected/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18-kube-api-access-dfqv2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:18.406392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.406352    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d659c741-776c-4043-a094-2d9e5b653777-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:18.407848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.407824    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx" pod "d659c741-776c-4043-a094-2d9e5b653777" (UID: "d659c741-776c-4043-a094-2d9e5b653777"). InnerVolumeSpecName "kube-api-access-vtffx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:18.508704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.508667    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtffx\" (UniqueName: \"kubernetes.io/projected/d659c741-776c-4043-a094-2d9e5b653777-kube-api-access-vtffx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:18.891351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.891310    5515 scope.go:122] "RemoveContainer" containerID="14059d10231567cd0b254b7a59dd743487eb77308e135255472b2ca11585f6c3"
Nov 03 03:30:18.896395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:18.896373    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" path="/var/lib/kubelet/pods/eb39043a-38b8-421b-825b-240ffbd44ec4/volumes"
Nov 03 03:30:19.020066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.020030    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:19.022044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.022022    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"
Nov 03 03:30:19.022183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.022020    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf" event={"ID":"7eea000d-fcfa-43cd-98d7-acb5f5ed9b18","Type":"ContainerDied","Data":"aacb1d8d23ef63ccd19a444cc76e4e3ad98dc5b719065fcf00981b2541f44a59"}
Nov 03 03:30:19.022183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.022143    5515 scope.go:122] "RemoveContainer" containerID="3f2e491591694884536c3b5201903fbdf7fd91453cdf7a728b9e3cb7049565c2"
Nov 03 03:30:19.023419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.023400    5515 generic.go:356] "Generic (PLEG): container finished" podID="d659c741-776c-4043-a094-2d9e5b653777" containerID="c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8" exitCode=0
Nov 03 03:30:19.023528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.023447    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"
Nov 03 03:30:19.023528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.023471    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x" event={"ID":"d659c741-776c-4043-a094-2d9e5b653777","Type":"ContainerDied","Data":"c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8"}
Nov 03 03:30:19.023528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.023512    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x" event={"ID":"d659c741-776c-4043-a094-2d9e5b653777","Type":"ContainerDied","Data":"5694a34d7811810c5bd06ae27a6b76db2816d65e0e897bf507a2b038f0ab34b3"}
Nov 03 03:30:19.028651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.028630    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerStarted","Data":"e26776b2b068fdd7af61982133c5b8d14b5c2b5278d49a34f99fa0b3765a72e9"}
Nov 03 03:30:19.033147 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.033132    5515 scope.go:122] "RemoveContainer" containerID="c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8"
Nov 03 03:30:19.042045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.042025    5515 scope.go:122] "RemoveContainer" containerID="c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8"
Nov 03 03:30:19.042305 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:19.042286    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8\": container with ID starting with c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8 not found: ID does not exist" containerID="c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8"
Nov 03 03:30:19.042349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.042311    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8"} err="failed to get container status \"c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8\": rpc error: code = NotFound desc = could not find container \"c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8\": container with ID starting with c660f32264e1772684db9a0fea36b67b4c39b207a9eaa98090f397471b7f5ac8 not found: ID does not exist"
Nov 03 03:30:19.166819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.164149    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"]
Nov 03 03:30:19.210912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.210774    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"]
Nov 03 03:30:19.214443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.214416    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d82a79-7fff5f74b9-wwpnf"]
Nov 03 03:30:19.348642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.348614    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:19.383424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.383403    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"]
Nov 03 03:30:19.423945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.423852    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh\" (UniqueName: \"kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh\") pod \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") "
Nov 03 03:30:19.423945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.423923    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca\") pod \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\" (UID: \"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a\") "
Nov 03 03:30:19.424587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.424521    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca" pod "9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" (UID: "9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:19.424812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.424781    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:19.426460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.426432    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh" pod "9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" (UID: "9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a"). InnerVolumeSpecName "kube-api-access-cdwmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:19.451763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.451734    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"]
Nov 03 03:30:19.485225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.485193    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d82a79-689dfccbc9-kk27x"]
Nov 03 03:30:19.526575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.526515    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdwmh\" (UniqueName: \"kubernetes.io/projected/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a-kube-api-access-cdwmh\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:19.539196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.539173    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:19.627068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.627034    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca\") pod \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") "
Nov 03 03:30:19.627259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.627136    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x\" (UniqueName: \"kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x\") pod \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\" (UID: \"bc36b093-09b4-43f3-88c4-294c70a0dbb5\") "
Nov 03 03:30:19.627447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.627420    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca" pod "bc36b093-09b4-43f3-88c4-294c70a0dbb5" (UID: "bc36b093-09b4-43f3-88c4-294c70a0dbb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:19.627903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.627879    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc36b093-09b4-43f3-88c4-294c70a0dbb5-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:19.630052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.630023    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x" pod "bc36b093-09b4-43f3-88c4-294c70a0dbb5" (UID: "bc36b093-09b4-43f3-88c4-294c70a0dbb5"). InnerVolumeSpecName "kube-api-access-d9m5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:19.730198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:19.730057    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9m5x\" (UniqueName: \"kubernetes.io/projected/bc36b093-09b4-43f3-88c4-294c70a0dbb5-kube-api-access-d9m5x\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:20.036806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.036727    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"
Nov 03 03:30:20.036806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.036732    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4" event={"ID":"9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a","Type":"ContainerDied","Data":"cd21c921c928b6983d60ad8659864d2d6b6de4d3643e9751ebb193978365b2d9"}
Nov 03 03:30:20.036806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.036802    5515 scope.go:122] "RemoveContainer" containerID="06c70787fc86aa36dcbfa0db05f85723a0aac9bbbe85a2d52ed184124c3704ce"
Nov 03 03:30:20.040377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.040354    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"
Nov 03 03:30:20.040504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.040352    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898" event={"ID":"bc36b093-09b4-43f3-88c4-294c70a0dbb5","Type":"ContainerDied","Data":"6363c39b95f22b3f438d1cac680b39a2ffd637ebf671e96f05b27b792f851b5c"}
Nov 03 03:30:20.042785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.042764    5515 generic.go:356] "Generic (PLEG): container finished" podID="8ce378bb-7357-46ae-8159-065d109e2baa" containerID="e26776b2b068fdd7af61982133c5b8d14b5c2b5278d49a34f99fa0b3765a72e9" exitCode=60
Nov 03 03:30:20.042896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.042832    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerDied","Data":"e26776b2b068fdd7af61982133c5b8d14b5c2b5278d49a34f99fa0b3765a72e9"}
Nov 03 03:30:20.043242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.043225    5515 scope.go:122] "RemoveContainer" containerID="e26776b2b068fdd7af61982133c5b8d14b5c2b5278d49a34f99fa0b3765a72e9"
Nov 03 03:30:20.043505 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:20.043485    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv_odh-ai-gateway-infra(8ce378bb-7357-46ae-8159-065d109e2baa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" podUID="8ce378bb-7357-46ae-8159-065d109e2baa"
Nov 03 03:30:20.044890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.044868    5515 generic.go:356] "Generic (PLEG): container finished" podID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerID="13c1210bfc263759fe7897945c42c1bd6f762e4aeed263c44c942c306eda7beb" exitCode=60
Nov 03 03:30:20.044995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.044940    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" event={"ID":"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a","Type":"ContainerDied","Data":"13c1210bfc263759fe7897945c42c1bd6f762e4aeed263c44c942c306eda7beb"}
Nov 03 03:30:20.045329 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.045310    5515 scope.go:122] "RemoveContainer" containerID="13c1210bfc263759fe7897945c42c1bd6f762e4aeed263c44c942c306eda7beb"
Nov 03 03:30:20.045605 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:20.045579    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k_odh-ai-gateway-infra(fd4df851-36f7-4c6d-8cb2-a4a6d89de81a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a"
Nov 03 03:30:20.049881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.049865    5515 scope.go:122] "RemoveContainer" containerID="993a65dd7bed21d379c8393223e26fdc79ee7d2608c2ef0f41380d42ab96bb82"
Nov 03 03:30:20.058905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.058890    5515 scope.go:122] "RemoveContainer" containerID="3f802a4e2e7afc46130f77eb5136f15e5718bf0da05d2202deb8af377865f22c"
Nov 03 03:30:20.071315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.071302    5515 scope.go:122] "RemoveContainer" containerID="14059d10231567cd0b254b7a59dd743487eb77308e135255472b2ca11585f6c3"
Nov 03 03:30:20.205208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.205171    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"]
Nov 03 03:30:20.209640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.209616    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4da1f5-29894610-fhrn4"]
Nov 03 03:30:20.340983 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.340905    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"]
Nov 03 03:30:20.558048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.558010    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"]
Nov 03 03:30:20.562808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.562772    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894610-cb898"]
Nov 03 03:30:20.896675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.896642    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" path="/var/lib/kubelet/pods/7eea000d-fcfa-43cd-98d7-acb5f5ed9b18/volumes"
Nov 03 03:30:20.897122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.897108    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" path="/var/lib/kubelet/pods/9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a/volumes"
Nov 03 03:30:20.897522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.897508    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" path="/var/lib/kubelet/pods/bc36b093-09b4-43f3-88c4-294c70a0dbb5/volumes"
Nov 03 03:30:20.898238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.898222    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d659c741-776c-4043-a094-2d9e5b653777" path="/var/lib/kubelet/pods/d659c741-776c-4043-a094-2d9e5b653777/volumes"
Nov 03 03:30:20.917416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:20.917388    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:20.987240 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:20.987203    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:30:21.231706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.231682    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:21.354401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.353954    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca\") pod \"8ce378bb-7357-46ae-8159-065d109e2baa\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") "
Nov 03 03:30:21.354401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.354078    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s\" (UniqueName: \"kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s\") pod \"8ce378bb-7357-46ae-8159-065d109e2baa\" (UID: \"8ce378bb-7357-46ae-8159-065d109e2baa\") "
Nov 03 03:30:21.354995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.354956    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca" pod "8ce378bb-7357-46ae-8159-065d109e2baa" (UID: "8ce378bb-7357-46ae-8159-065d109e2baa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:21.358420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.358390    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s" pod "8ce378bb-7357-46ae-8159-065d109e2baa" (UID: "8ce378bb-7357-46ae-8159-065d109e2baa"). InnerVolumeSpecName "kube-api-access-g2z6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:21.455243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.455157    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"]
Nov 03 03:30:21.456197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.456171    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ce378bb-7357-46ae-8159-065d109e2baa-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:21.456280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.456200    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2z6s\" (UniqueName: \"kubernetes.io/projected/8ce378bb-7357-46ae-8159-065d109e2baa-kube-api-access-g2z6s\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:21.634960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.634935    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:21.761681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.761598    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp\" (UniqueName: \"kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp\") pod \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") "
Nov 03 03:30:21.761859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.761719    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca\") pod \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\" (UID: \"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a\") "
Nov 03 03:30:21.762124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.762097    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca" pod "fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" (UID: "fd4df851-36f7-4c6d-8cb2-a4a6d89de81a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:21.762342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.762324    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:21.764738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.764426    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp" pod "fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" (UID: "fd4df851-36f7-4c6d-8cb2-a4a6d89de81a"). InnerVolumeSpecName "kube-api-access-9xxpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:21.863353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:21.863315    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xxpp\" (UniqueName: \"kubernetes.io/projected/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a-kube-api-access-9xxpp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:22.064432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.064341    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"
Nov 03 03:30:22.064432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.064369    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv" event={"ID":"8ce378bb-7357-46ae-8159-065d109e2baa","Type":"ContainerDied","Data":"067305423f40c376c06719d984640722f8c461c2bf8bebddb38dcde4f2d0755b"}
Nov 03 03:30:22.064432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.064430    5515 scope.go:122] "RemoveContainer" containerID="e26776b2b068fdd7af61982133c5b8d14b5c2b5278d49a34f99fa0b3765a72e9"
Nov 03 03:30:22.066634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.066612    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"
Nov 03 03:30:22.066755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.066629    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k" event={"ID":"fd4df851-36f7-4c6d-8cb2-a4a6d89de81a","Type":"ContainerDied","Data":"fff22ed0338f982f3a1087dcd22530356d36e45e1ab29094b5e1a7d495346275"}
Nov 03 03:30:22.077229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.077211    5515 scope.go:122] "RemoveContainer" containerID="13c1210bfc263759fe7897945c42c1bd6f762e4aeed263c44c942c306eda7beb"
Nov 03 03:30:22.236974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.236940    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"]
Nov 03 03:30:22.287464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.287427    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-9902a0c2-29894610-jkbmv"]
Nov 03 03:30:22.353537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.353456    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"]
Nov 03 03:30:22.357280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.357255    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894610-7zq5k"]
Nov 03 03:30:22.895996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.895962    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" path="/var/lib/kubelet/pods/8ce378bb-7357-46ae-8159-065d109e2baa/volumes"
Nov 03 03:30:22.896641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:22.896624    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" path="/var/lib/kubelet/pods/fd4df851-36f7-4c6d-8cb2-a4a6d89de81a/volumes"
Nov 03 03:30:23.858901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:23.858873    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:24.029578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:24.026011    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:24.173633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:24.170788    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:28.939883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:28.939829    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:29.068941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:29.068660    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"]
Nov 03 03:30:29.104974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:29.104942    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6326d17-vntn7"]
Nov 03 03:30:30.009132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:30.009103    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:30:30.895998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:30.895960    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202b7306-df66-41a3-8b26-207d2eab763e" path="/var/lib/kubelet/pods/202b7306-df66-41a3-8b26-207d2eab763e/volumes"
Nov 03 03:30:33.830255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:33.830215    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:38.084610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:38.084528    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:39.928930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:39.928890    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:40.735045 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:40.735003    5515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bbf043_42ca_4359_92b7_fb8ec834da65.slice/crio-28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bbf043_42ca_4359_92b7_fb8ec834da65.slice/crio-conmon-28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@8-22281-0.service\": RecentStats: unable to find data in memory cache]"
Nov 03 03:30:41.304720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:41.304683    5515 generic.go:356] "Generic (PLEG): container finished" podID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerID="28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d" exitCode=0
Nov 03 03:30:41.305261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:41.304730    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" event={"ID":"a4bbf043-42ca-4359-92b7-fb8ec834da65","Type":"ContainerDied","Data":"28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d"}
Nov 03 03:30:41.305261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:41.305197    5515 scope.go:122] "RemoveContainer" containerID="28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d"
Nov 03 03:30:41.307347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:41.307303    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:41.307472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:41.307404    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:41.307472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:41.307457    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:42.315253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:42.315219    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" event={"ID":"a4bbf043-42ca-4359-92b7-fb8ec834da65","Type":"ContainerStarted","Data":"9963df0cece920f5d612b8ae406e3e3d24c558845049c0a67f2168160ed048c0"}
Nov 03 03:30:43.057862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:43.057828    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:30:43.058047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:43.058026    5515 patch_prober.go:29] interesting pod/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 03:30:43.058108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:43.058087    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 03:30:44.060574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:44.058144    5515 patch_prober.go:29] interesting pod/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 03:30:44.060574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:44.058222    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 03:30:44.819161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:44.819117    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:45.036266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:45.032019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:45.067582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:45.060067    5515 patch_prober.go:29] interesting pod/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 03:30:45.067582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:45.060146    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 03:30:45.190570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:45.190183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:46.057861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:46.057829    5515 patch_prober.go:29] interesting pod/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 03:30:46.058043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:46.057888    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 03:30:46.205576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:46.194113    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:46.890276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:46.890240    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:47.057959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:47.057925    5515 patch_prober.go:29] interesting pod/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 03:30:47.058139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:47.057991    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 03:30:47.352744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:47.352524    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"]
Nov 03 03:30:47.353218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:47.352802    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy" containerID="cri-o://31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b" gracePeriod=30
Nov 03 03:30:47.427243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:47.425528    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:47.575081 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:47.575044    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:48.046944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:48.046905    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:30:48.068878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:48.068844    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:30:48.156384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:48.156354    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:48.446269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:48.446243    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:30:50.169194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.169160    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:50.630094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630062    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"]
Nov 03 03:30:50.630834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630813    5515 cpu_manager.go:414] "containerMap: removing container" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.630834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630834    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630865    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630873    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630887    5515 cpu_manager.go:414] "containerMap: removing container" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630894    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630909    5515 cpu_manager.go:414] "containerMap: removing container" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630917    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630930    5515 cpu_manager.go:414] "containerMap: removing container" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630936    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630946    5515 cpu_manager.go:414] "containerMap: removing container" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630953    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630966    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630973    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630993    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.630999    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631011    5515 cpu_manager.go:414] "containerMap: removing container" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:30:50.631016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631017    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631029    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631036    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631061    5515 cpu_manager.go:414] "containerMap: removing container" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631069    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631083    5515 cpu_manager.go:414] "containerMap: removing container" podUID="7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" containerName="payload-processing"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631090    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" containerName="payload-processing"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631103    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631110    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631124    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d659c741-776c-4043-a094-2d9e5b653777" containerName="payload-pre-processing"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631131    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d659c741-776c-4043-a094-2d9e5b653777" containerName="payload-pre-processing"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631331    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631346    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631356    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631368    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631380    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631388    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631398    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631406    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631416    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631424    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631432    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea000d-fcfa-43cd-98d7-acb5f5ed9b18" containerName="payload-processing"
Nov 03 03:30:50.631851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.631442    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d659c741-776c-4043-a094-2d9e5b653777" containerName="payload-pre-processing"
Nov 03 03:30:50.635719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.635700    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.715915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.715875    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716068    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hr5\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716188 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716170    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716220    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716253    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716364    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716490    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716607    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.716739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.716682    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818275    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818326    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818354    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818388    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818457    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818481    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818512    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818624    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818728    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hr5\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.818833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.818744    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.819134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.819112    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.819305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.819285    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.819575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.819534    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.819651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.819540    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.820961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.820937    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.821266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.821246    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.877366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.877332    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"]
Nov 03 03:30:50.880776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.880706    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hr5\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.883174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.883143    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token\") pod \"e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") " pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.952419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:50.952382    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:50.992290 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:50.992248    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:30:51.389123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.389094    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"]
Nov 03 03:30:51.390532 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:30:51.390510    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a74f5d_e683_4502_8550_241b634591db.slice/crio-0b2c21ab6080e5c2fc29f7d2611e18de5c77e8d97c8675759867f14fa2ed64b8 WatchSource:0}: Error finding container 0b2c21ab6080e5c2fc29f7d2611e18de5c77e8d97c8675759867f14fa2ed64b8: Status 404 returned error can't find the container with id 0b2c21ab6080e5c2fc29f7d2611e18de5c77e8d97c8675759867f14fa2ed64b8
Nov 03 03:30:51.393768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.393737    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:51.393835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.393804    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:51.393835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.393832    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:51.471909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.471875    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" event={"ID":"01a74f5d-e683-4502-8550-241b634591db","Type":"ContainerStarted","Data":"ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd"}
Nov 03 03:30:51.472089 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.471932    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" event={"ID":"01a74f5d-e683-4502-8550-241b634591db","Type":"ContainerStarted","Data":"0b2c21ab6080e5c2fc29f7d2611e18de5c77e8d97c8675759867f14fa2ed64b8"}
Nov 03 03:30:51.597803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.597751    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podStartSLOduration=1.597737692 podStartE2EDuration="1.597737692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:30:51.595744779 +0000 UTC m=+1742.826637197" watchObservedRunningTime="2026-11-03 03:30:51.597737692 +0000 UTC m=+1742.828630109"
Nov 03 03:30:51.953222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:51.953181    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:52.728112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.728081    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:30:52.786705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.786652    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-742fn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:30:52.786928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.786728    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:30:52.790036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.789985    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"]
Nov 03 03:30:52.854162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854119    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854194    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854239    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854279    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854361    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854400    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:30:52.854563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854417    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854495    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854581    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:52.854774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854590    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854665    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo\") pod \"45285d6d-5238-4608-ad26-49df796027b2\" (UID: \"45285d6d-5238-4608-ad26-49df796027b2\") "
Nov 03 03:30:52.854774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854688    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:30:52.854980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854777    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:30:52.854980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.854889    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:30:52.855401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.855344    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.855401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.855364    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.855401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.855377    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.855401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.855390    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45285d6d-5238-4608-ad26-49df796027b2-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.855401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.855398    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.857369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.857317    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:30:52.857802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.857776    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:52.860928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.860905    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "kube-api-access-97lww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:52.861032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.861001    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy" pod "45285d6d-5238-4608-ad26-49df796027b2" (UID: "45285d6d-5238-4608-ad26-49df796027b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:30:52.916206 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.916171    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"]
Nov 03 03:30:52.954608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.954510    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 03:30:52.954608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.954586    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 03:30:52.957272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.957247    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45285d6d-5238-4608-ad26-49df796027b2-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.957348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.957283    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.957348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.957303    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lww\" (UniqueName: \"kubernetes.io/projected/45285d6d-5238-4608-ad26-49df796027b2-kube-api-access-97lww\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:52.957348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:52.957319    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45285d6d-5238-4608-ad26-49df796027b2-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:53.144971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.144932    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:53.289413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.289321    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:53.512820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.512774    5515 generic.go:356] "Generic (PLEG): container finished" podID="45285d6d-5238-4608-ad26-49df796027b2" containerID="31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b" exitCode=0
Nov 03 03:30:53.512820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.512824    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"
Nov 03 03:30:53.513056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.512856    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" event={"ID":"45285d6d-5238-4608-ad26-49df796027b2","Type":"ContainerDied","Data":"31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b"}
Nov 03 03:30:53.513056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.512916    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96" event={"ID":"45285d6d-5238-4608-ad26-49df796027b2","Type":"ContainerDied","Data":"05a3826ec560d37a1e4545c90ac447bf3f82d3171732e4a3336b166fc7906380"}
Nov 03 03:30:53.513056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.512941    5515 scope.go:122] "RemoveContainer" containerID="31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b"
Nov 03 03:30:53.524627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.524562    5515 scope.go:122] "RemoveContainer" containerID="422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"
Nov 03 03:30:53.534306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.534279    5515 scope.go:122] "RemoveContainer" containerID="31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b"
Nov 03 03:30:53.534645 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:53.534626    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b\": container with ID starting with 31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b not found: ID does not exist" containerID="31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b"
Nov 03 03:30:53.534735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.534658    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b"} err="failed to get container status \"31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b\": rpc error: code = NotFound desc = could not find container \"31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b\": container with ID starting with 31d72f7f4a9ff6f90e38164cb3af2fc87658ff3a68bd4ffb7fc6c5269dcb612b not found: ID does not exist"
Nov 03 03:30:53.534735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.534685    5515 scope.go:122] "RemoveContainer" containerID="422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"
Nov 03 03:30:53.534949 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:30:53.534929    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954\": container with ID starting with 422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954 not found: ID does not exist" containerID="422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"
Nov 03 03:30:53.535002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.534958    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954"} err="failed to get container status \"422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954\": rpc error: code = NotFound desc = could not find container \"422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954\": container with ID starting with 422c903ca15a4eff3d1abdbd41c8d2dd05476151df95eea7560cb54d82e8d954 not found: ID does not exist"
Nov 03 03:30:53.693778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.693740    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"]
Nov 03 03:30:53.697353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.697326    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d82a79-openshift-default-8454f5696c-fwt96"]
Nov 03 03:30:53.953517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.953437    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 03:30:53.953517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:53.953493    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 03:30:54.378588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:54.377482    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:54.842592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:54.833352    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:54.953725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:54.953683    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45285d6d-5238-4608-ad26-49df796027b2" path="/var/lib/kubelet/pods/45285d6d-5238-4608-ad26-49df796027b2/volumes"
Nov 03 03:30:54.954201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:54.953722    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 03:30:54.954201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:54.953796    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 03:30:55.053291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.053254    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:55.238342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.237873    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"]
Nov 03 03:30:55.347164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.347134    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"]
Nov 03 03:30:55.445106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.444761    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:55.589207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.589105    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:55.955035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.954997    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 03:30:55.963801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:55.955580    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 03:30:56.608618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.608589    5515 generic.go:356] "Generic (PLEG): container finished" podID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerID="ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6" exitCode=0
Nov 03 03:30:56.608788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.608593    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" event={"ID":"036c8e7a-11e5-4fb6-9471-23698540e2ca","Type":"ContainerDied","Data":"ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"}
Nov 03 03:30:56.609091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.609077    5515 scope.go:122] "RemoveContainer" containerID="ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"
Nov 03 03:30:56.610514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.610482    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:56.610629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.610579    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:56.610629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.610618    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:30:56.972634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.956376    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 03:30:56.972634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.956448    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 03:30:56.997575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:56.988120    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:57.576572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.572596    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:30:57.626572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.626113    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" event={"ID":"036c8e7a-11e5-4fb6-9471-23698540e2ca","Type":"ContainerStarted","Data":"2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e"}
Nov 03 03:30:57.791574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.790613    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g" podUID="6b8d5e33-b51b-4707-8eda-88fa0881d9b6" containerName="payload-processing" containerID="cri-o://f5b7cbf731608d7e02fad39a2c8537611e80bf5e16eba6c2aff61cd90ee12bff" gracePeriod=25
Nov 03 03:30:57.916942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.916897    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d" podUID="93085299-7eb4-495f-ad03-f95594fc176a" containerName="payload-pre-processing" containerID="cri-o://0d069af0db603245579925af6d7bf94068d8ea10b18c2475684a924a1efa55dc" gracePeriod=25
Nov 03 03:30:57.953066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.953023    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 03 03:30:57.953253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.953100    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 03 03:30:58.002577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:57.998903    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:30:58.960825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:58.957971    5515 generic.go:356] "Generic (PLEG): container finished" podID="93085299-7eb4-495f-ad03-f95594fc176a" containerID="0d069af0db603245579925af6d7bf94068d8ea10b18c2475684a924a1efa55dc" exitCode=0
Nov 03 03:30:58.960825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:58.960418    5515 generic.go:356] "Generic (PLEG): container finished" podID="6b8d5e33-b51b-4707-8eda-88fa0881d9b6" containerID="f5b7cbf731608d7e02fad39a2c8537611e80bf5e16eba6c2aff61cd90ee12bff" exitCode=0
Nov 03 03:30:58.985752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:58.985712    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d" event={"ID":"93085299-7eb4-495f-ad03-f95594fc176a","Type":"ContainerDied","Data":"0d069af0db603245579925af6d7bf94068d8ea10b18c2475684a924a1efa55dc"}
Nov 03 03:30:58.985979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:58.985955    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g" event={"ID":"6b8d5e33-b51b-4707-8eda-88fa0881d9b6","Type":"ContainerDied","Data":"f5b7cbf731608d7e02fad39a2c8537611e80bf5e16eba6c2aff61cd90ee12bff"}
Nov 03 03:30:58.991798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:58.991768    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-742fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 03:30:58.999669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:58.996498    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 03:30:59.011525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.011504    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:30:59.106388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.106354    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn\" (UniqueName: \"kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn\") pod \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") "
Nov 03 03:30:59.106702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.106681    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume\") pod \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\" (UID: \"6b8d5e33-b51b-4707-8eda-88fa0881d9b6\") "
Nov 03 03:30:59.109210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.109180    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume" pod "6b8d5e33-b51b-4707-8eda-88fa0881d9b6" (UID: "6b8d5e33-b51b-4707-8eda-88fa0881d9b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:59.115097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.115068    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn" pod "6b8d5e33-b51b-4707-8eda-88fa0881d9b6" (UID: "6b8d5e33-b51b-4707-8eda-88fa0881d9b6"). InnerVolumeSpecName "kube-api-access-fghqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:59.185588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.160836    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:59.185588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.171217    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:59.185588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.181892    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:30:59.209358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.209325    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fghqn\" (UniqueName: \"kubernetes.io/projected/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-kube-api-access-fghqn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:59.209498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.209368    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b8d5e33-b51b-4707-8eda-88fa0881d9b6-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:59.215958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.215887    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:30:59.310844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.310808    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume\") pod \"93085299-7eb4-495f-ad03-f95594fc176a\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") "
Nov 03 03:30:59.311011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.310938    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5\" (UniqueName: \"kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5\") pod \"93085299-7eb4-495f-ad03-f95594fc176a\" (UID: \"93085299-7eb4-495f-ad03-f95594fc176a\") "
Nov 03 03:30:59.311092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.311072    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume" pod "93085299-7eb4-495f-ad03-f95594fc176a" (UID: "93085299-7eb4-495f-ad03-f95594fc176a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:30:59.311743 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.311720    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93085299-7eb4-495f-ad03-f95594fc176a-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:59.313247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.313225    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5" pod "93085299-7eb4-495f-ad03-f95594fc176a" (UID: "93085299-7eb4-495f-ad03-f95594fc176a"). InnerVolumeSpecName "kube-api-access-tlgd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:30:59.413264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.413230    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlgd5\" (UniqueName: \"kubernetes.io/projected/93085299-7eb4-495f-ad03-f95594fc176a-kube-api-access-tlgd5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:30:59.990914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.990879    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-742fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 03:30:59.991114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.990935    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 03:30:59.992961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.992942    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"
Nov 03 03:30:59.993079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.992987    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g" event={"ID":"6b8d5e33-b51b-4707-8eda-88fa0881d9b6","Type":"ContainerDied","Data":"c5ddd64a8dd6dfc97a681e978e7b831711dd3677243f8e87277749983324f3bb"}
Nov 03 03:30:59.993079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.993045    5515 scope.go:122] "RemoveContainer" containerID="f5b7cbf731608d7e02fad39a2c8537611e80bf5e16eba6c2aff61cd90ee12bff"
Nov 03 03:30:59.994678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.994644    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"
Nov 03 03:30:59.994791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:30:59.994684    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d" event={"ID":"93085299-7eb4-495f-ad03-f95594fc176a","Type":"ContainerDied","Data":"463cf80befc5ad3e3754b561a9517cc0003b75cdd98faa2521a12bce3ad5472f"}
Nov 03 03:31:00.004155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.004134    5515 scope.go:122] "RemoveContainer" containerID="0d069af0db603245579925af6d7bf94068d8ea10b18c2475684a924a1efa55dc"
Nov 03 03:31:00.186710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.186676    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"]
Nov 03 03:31:00.190218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.190192    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b562f5-587f99f88c-crd6d"]
Nov 03 03:31:00.238289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.238207    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f" podUID="1baaede1-f5ca-427b-a294-9718bf6481d4" containerName="payload-processing" containerID="cri-o://766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1" gracePeriod=25
Nov 03 03:31:00.299395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.299358    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"]
Nov 03 03:31:00.303656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.303632    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b562f5-cc5fb68f5-kk99g"]
Nov 03 03:31:00.348285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.348243    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw" podUID="f61737ad-a4a3-4d6e-b163-99e6cbda21c7" containerName="payload-pre-processing" containerID="cri-o://c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953" gracePeriod=25
Nov 03 03:31:00.392706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.390680    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:00.608769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.608737    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:31:00.653610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.653581    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:31:00.731132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.731097    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume\") pod \"1baaede1-f5ca-427b-a294-9718bf6481d4\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") "
Nov 03 03:31:00.731302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.731271    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72\" (UniqueName: \"kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72\") pod \"1baaede1-f5ca-427b-a294-9718bf6481d4\" (UID: \"1baaede1-f5ca-427b-a294-9718bf6481d4\") "
Nov 03 03:31:00.731529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.731502    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume" pod "1baaede1-f5ca-427b-a294-9718bf6481d4" (UID: "1baaede1-f5ca-427b-a294-9718bf6481d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:00.731774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.731758    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1baaede1-f5ca-427b-a294-9718bf6481d4-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:00.733526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.733509    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72" pod "1baaede1-f5ca-427b-a294-9718bf6481d4" (UID: "1baaede1-f5ca-427b-a294-9718bf6481d4"). InnerVolumeSpecName "kube-api-access-87j72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:00.832363 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.832283    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume\") pod \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") "
Nov 03 03:31:00.832517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.832485    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb\" (UniqueName: \"kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb\") pod \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\" (UID: \"f61737ad-a4a3-4d6e-b163-99e6cbda21c7\") "
Nov 03 03:31:00.832680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.832657    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume" pod "f61737ad-a4a3-4d6e-b163-99e6cbda21c7" (UID: "f61737ad-a4a3-4d6e-b163-99e6cbda21c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:00.833171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.833154    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:00.833212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.833177    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87j72\" (UniqueName: \"kubernetes.io/projected/1baaede1-f5ca-427b-a294-9718bf6481d4-kube-api-access-87j72\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:00.834740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.834723    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb" pod "f61737ad-a4a3-4d6e-b163-99e6cbda21c7" (UID: "f61737ad-a4a3-4d6e-b163-99e6cbda21c7"). InnerVolumeSpecName "kube-api-access-xnjkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:00.900941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.900906    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8d5e33-b51b-4707-8eda-88fa0881d9b6" path="/var/lib/kubelet/pods/6b8d5e33-b51b-4707-8eda-88fa0881d9b6/volumes"
Nov 03 03:31:00.901390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.901372    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93085299-7eb4-495f-ad03-f95594fc176a" path="/var/lib/kubelet/pods/93085299-7eb4-495f-ad03-f95594fc176a/volumes"
Nov 03 03:31:00.935040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.935004    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnjkb\" (UniqueName: \"kubernetes.io/projected/f61737ad-a4a3-4d6e-b163-99e6cbda21c7-kube-api-access-xnjkb\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:00.990856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.990825    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-742fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 03:31:00.991002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:00.990884    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 03:31:01.002734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.002703    5515 generic.go:356] "Generic (PLEG): container finished" podID="1baaede1-f5ca-427b-a294-9718bf6481d4" containerID="766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1" exitCode=0
Nov 03 03:31:01.002871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.002759    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"
Nov 03 03:31:01.002871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.002792    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f" event={"ID":"1baaede1-f5ca-427b-a294-9718bf6481d4","Type":"ContainerDied","Data":"766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1"}
Nov 03 03:31:01.002871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.002839    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f" event={"ID":"1baaede1-f5ca-427b-a294-9718bf6481d4","Type":"ContainerDied","Data":"72130971a7fc16ac8b61a80e07f1c5e68abbef7c18b374ca3968235b277db281"}
Nov 03 03:31:01.002871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.002866    5515 scope.go:122] "RemoveContainer" containerID="766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1"
Nov 03 03:31:01.006392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.006372    5515 generic.go:356] "Generic (PLEG): container finished" podID="f61737ad-a4a3-4d6e-b163-99e6cbda21c7" containerID="c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953" exitCode=0
Nov 03 03:31:01.006502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.006425    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"
Nov 03 03:31:01.006502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.006438    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw" event={"ID":"f61737ad-a4a3-4d6e-b163-99e6cbda21c7","Type":"ContainerDied","Data":"c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953"}
Nov 03 03:31:01.006502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.006476    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw" event={"ID":"f61737ad-a4a3-4d6e-b163-99e6cbda21c7","Type":"ContainerDied","Data":"578fc949fe1363ad35b057423f9d0169bcc2cd2d3c376dcccee79b326072816f"}
Nov 03 03:31:01.014078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.014061    5515 scope.go:122] "RemoveContainer" containerID="766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1"
Nov 03 03:31:01.014345 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:01.014328    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1\": container with ID starting with 766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1 not found: ID does not exist" containerID="766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1"
Nov 03 03:31:01.014394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.014352    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1"} err="failed to get container status \"766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1\": rpc error: code = NotFound desc = could not find container \"766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1\": container with ID starting with 766720e7f4ff5ed9adba2bf43596697e49be9a65b6c2d428e3522b2b7f900ef1 not found: ID does not exist"
Nov 03 03:31:01.014394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.014369    5515 scope.go:122] "RemoveContainer" containerID="c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953"
Nov 03 03:31:01.023356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.023340    5515 scope.go:122] "RemoveContainer" containerID="c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953"
Nov 03 03:31:01.023640 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:01.023618    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953\": container with ID starting with c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953 not found: ID does not exist" containerID="c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953"
Nov 03 03:31:01.023711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.023647    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953"} err="failed to get container status \"c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953\": rpc error: code = NotFound desc = could not find container \"c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953\": container with ID starting with c9bbb91846a30d777f672217ba857d89b3cc7dc7ba2c9af634a582b871913953 not found: ID does not exist"
Nov 03 03:31:01.204213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.204164    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"]
Nov 03 03:31:01.208439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.208408    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4da1f5-fcf7c85ff-splhw"]
Nov 03 03:31:01.251993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.251956    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:01.306624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.306587    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"]
Nov 03 03:31:01.311472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.311437    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4da1f5-8488b4fddc-rcj2f"]
Nov 03 03:31:01.400492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.396029    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:01.991013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.990980    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-742fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 03:31:01.991164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:01.991033    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 03:31:02.185577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:02.184365    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:02.792462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:02.792429    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:02.941263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:02.941226    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1baaede1-f5ca-427b-a294-9718bf6481d4" path="/var/lib/kubelet/pods/1baaede1-f5ca-427b-a294-9718bf6481d4/volumes"
Nov 03 03:31:02.941840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:02.941819    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61737ad-a4a3-4d6e-b163-99e6cbda21c7" path="/var/lib/kubelet/pods/f61737ad-a4a3-4d6e-b163-99e6cbda21c7/volumes"
Nov 03 03:31:02.991596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:02.990586    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-742fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 03 03:31:02.991596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:02.990662    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 03 03:31:03.304356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:03.297869    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:03.304356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:03.298103    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"]
Nov 03 03:31:03.304356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:03.298363    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" containerID="cri-o://3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70" gracePeriod=30
Nov 03 03:31:03.995955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:03.995927    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:31:03.996353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:03.996330    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:31:03.997118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:03.997094    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:31:05.513680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:05.513639    5515 patch_prober.go:29] interesting pod/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:31:05.514062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:05.513711    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:31:07.000576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:06.989146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:07.956805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:07.956772    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"]
Nov 03 03:31:07.957216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:07.957186    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy" containerID="cri-o://ec88d68992a90524c670f18793d1ada4eb8b7e71d2a35b6573ef3d40bbb58476" gracePeriod=30
Nov 03 03:31:08.006929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.006898    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"]
Nov 03 03:31:08.007325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.007173    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy" containerID="cri-o://7e79f283a52fdb436935e49eb595d023e16b76035b04319fafaaf8b1d8cc1062" gracePeriod=30
Nov 03 03:31:08.010560 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.010506    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"]
Nov 03 03:31:08.010793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.010767    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" containerID="cri-o://fe69845a8ac5286473ce1b281b955ffef0117d44a1817919e5d81a52c2c4e22e" gracePeriod=30
Nov 03 03:31:08.056695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.056660    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"]
Nov 03 03:31:08.056887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.056844    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy" containerID="cri-o://a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977" gracePeriod=30
Nov 03 03:31:08.439949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.439912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:08.592835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.592807    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:31:08.653669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.653642    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.653820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.653691    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.653820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.653718    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.653820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.653809    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.653854    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654019    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654086    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654119    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654162    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654190    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654215    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs\") pod \"1a614825-5146-44d1-8753-a72927a9e298\" (UID: \"1a614825-5146-44d1-8753-a72927a9e298\") "
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654190    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:08.654440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654301    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:08.654920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.654643    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:08.655654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.655632    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.655796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.655781    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.655900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.655888    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.655999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.655977    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a614825-5146-44d1-8753-a72927a9e298-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.656126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.656004    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.657440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.657406    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:08.657542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.657481    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:08.657542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.657485    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:08.657542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.657506    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2" pod "1a614825-5146-44d1-8753-a72927a9e298" (UID: "1a614825-5146-44d1-8753-a72927a9e298"). InnerVolumeSpecName "kube-api-access-jbfz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:08.707822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.707790    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:08.847674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.791487    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbfz2\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-kube-api-access-jbfz2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.847921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.847896    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a614825-5146-44d1-8753-a72927a9e298-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.848018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.848006    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a614825-5146-44d1-8753-a72927a9e298-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.848112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.848101    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a614825-5146-44d1-8753-a72927a9e298-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:08.851408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:08.851367    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:09.140627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.140592    5515 generic.go:356] "Generic (PLEG): container finished" podID="1a614825-5146-44d1-8753-a72927a9e298" containerID="3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70" exitCode=0
Nov 03 03:31:09.141056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.140675    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"
Nov 03 03:31:09.141056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.140709    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" event={"ID":"1a614825-5146-44d1-8753-a72927a9e298","Type":"ContainerDied","Data":"3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70"}
Nov 03 03:31:09.141056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.140764    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45" event={"ID":"1a614825-5146-44d1-8753-a72927a9e298","Type":"ContainerDied","Data":"9e59a861e45efd3071a55ac8595d9107023efa7a6fc80b11c18a2d74ff3e484e"}
Nov 03 03:31:09.141056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.140786    5515 scope.go:122] "RemoveContainer" containerID="3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70"
Nov 03 03:31:09.150866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.150844    5515 scope.go:122] "RemoveContainer" containerID="c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"
Nov 03 03:31:09.160467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.160442    5515 scope.go:122] "RemoveContainer" containerID="3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70"
Nov 03 03:31:09.160763 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:09.160745    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70\": container with ID starting with 3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70 not found: ID does not exist" containerID="3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70"
Nov 03 03:31:09.160820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.160771    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70"} err="failed to get container status \"3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70\": rpc error: code = NotFound desc = could not find container \"3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70\": container with ID starting with 3df0663551c3e9fcad07eefcc4c6e2e76583173621c84f69efb44254cb62af70 not found: ID does not exist"
Nov 03 03:31:09.160820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.160789    5515 scope.go:122] "RemoveContainer" containerID="c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"
Nov 03 03:31:09.161032 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:09.161014    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7\": container with ID starting with c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7 not found: ID does not exist" containerID="c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"
Nov 03 03:31:09.161075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.161037    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7"} err="failed to get container status \"c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7\": rpc error: code = NotFound desc = could not find container \"c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7\": container with ID starting with c51c6a63273965567bb2b009240d9f37b0f4f6744c46f93fba1b688ce1f1a6f7 not found: ID does not exist"
Nov 03 03:31:09.309344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.309312    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"]
Nov 03 03:31:09.360031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:09.359991    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4da1f5-openshift-default-6899df7885-4dq45"]
Nov 03 03:31:10.895703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:10.895672    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a614825-5146-44d1-8753-a72927a9e298" path="/var/lib/kubelet/pods/1a614825-5146-44d1-8753-a72927a9e298/volumes"
Nov 03 03:31:11.604868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:11.604837    5515 patch_prober.go:29] interesting pod/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:31:11.605050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:11.604900    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:31:12.414980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:12.414940    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:13.085413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.085381    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:13.208042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.208009    5515 generic.go:356] "Generic (PLEG): container finished" podID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerID="7e79f283a52fdb436935e49eb595d023e16b76035b04319fafaaf8b1d8cc1062" exitCode=0
Nov 03 03:31:13.208200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.208177    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" event={"ID":"442db502-38c6-4dfa-b9e3-1e03b0461ee0","Type":"ContainerDied","Data":"7e79f283a52fdb436935e49eb595d023e16b76035b04319fafaaf8b1d8cc1062"}
Nov 03 03:31:13.209895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.209870    5515 generic.go:356] "Generic (PLEG): container finished" podID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerID="fe69845a8ac5286473ce1b281b955ffef0117d44a1817919e5d81a52c2c4e22e" exitCode=0
Nov 03 03:31:13.210012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.209985    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" event={"ID":"64709aa4-be02-4056-a5a7-756f4c6bad1b","Type":"ContainerDied","Data":"fe69845a8ac5286473ce1b281b955ffef0117d44a1817919e5d81a52c2c4e22e"}
Nov 03 03:31:13.212318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.211839    5515 generic.go:356] "Generic (PLEG): container finished" podID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerID="ec88d68992a90524c670f18793d1ada4eb8b7e71d2a35b6573ef3d40bbb58476" exitCode=0
Nov 03 03:31:13.212318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.211967    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" event={"ID":"6388c0bd-7d55-4dd5-b203-ab82a6102a25","Type":"ContainerDied","Data":"ec88d68992a90524c670f18793d1ada4eb8b7e71d2a35b6573ef3d40bbb58476"}
Nov 03 03:31:13.236209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.236067    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:13.397691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.397176    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:31:13.463031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.463004    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:31:13.473887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.473851    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:31:13.479947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.479920    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:31:13.517860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.517779    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.517860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.517819    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.517860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.517855    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.517897    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.517932    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.517996    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518019    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518126    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518229    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs\") pod \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\" (UID: \"6388c0bd-7d55-4dd5-b203-ab82a6102a25\") "
Nov 03 03:31:13.518335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518120    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.518335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518219    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.518538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518335    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.518538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518365    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:13.518684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.518526    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.519273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.519187    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.519273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.519212    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.519273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.519228    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.519273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.519242    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.519273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.519257    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.520261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.520237    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.520419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.520399    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:13.520419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.520406    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.520674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.520659    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m" pod "6388c0bd-7d55-4dd5-b203-ab82a6102a25" (UID: "6388c0bd-7d55-4dd5-b203-ab82a6102a25"). InnerVolumeSpecName "kube-api-access-v6l8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.620091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620021    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620085    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.620352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620132    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.620352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620167    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.620456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620343    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620404    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620435    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.620672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620462    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620490    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.620672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620523    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.620672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620575    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.620672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620630    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620694    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.620934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620710    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.620934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620741    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.620934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620827    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620887    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.620934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620914    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620952    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.620989    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621050    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621078    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621118    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621160    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.621254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621214    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.621825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621275    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket\") pod \"e7ed1963-4270-4551-a719-175e5ca80aa7\" (UID: \"e7ed1963-4270-4551-a719-175e5ca80aa7\") "
Nov 03 03:31:13.621825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621303    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.621825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621755    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert\") pod \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\" (UID: \"442db502-38c6-4dfa-b9e3-1e03b0461ee0\") "
Nov 03 03:31:13.621825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.621805    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs\") pod \"64709aa4-be02-4056-a5a7-756f4c6bad1b\" (UID: \"64709aa4-be02-4056-a5a7-756f4c6bad1b\") "
Nov 03 03:31:13.622667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622640    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.622775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622670    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.622775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622687    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.622775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622701    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.622775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622718    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.622775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622732    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6l8m\" (UniqueName: \"kubernetes.io/projected/6388c0bd-7d55-4dd5-b203-ab82a6102a25-kube-api-access-v6l8m\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.623061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.622950    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.623159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623133    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:13.623227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623147    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.623227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623141    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.623391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623347    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.623648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623624    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.623742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623501    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:13.623742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623715    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.623871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623850    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.623971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.623948    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.624180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.624158    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.624269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.624245    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:13.624470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.624445    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:13.624738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.624715    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.624950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.624894    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.627345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.626136    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.627345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.626366    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.627345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.627007    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "kube-api-access-svvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.627798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.627493    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.627798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.627597    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.627924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.627813    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:13.629455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.629430    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo" pod "e7ed1963-4270-4551-a719-175e5ca80aa7" (UID: "e7ed1963-4270-4551-a719-175e5ca80aa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:13.629455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.629438    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6" pod "64709aa4-be02-4056-a5a7-756f4c6bad1b" (UID: "64709aa4-be02-4056-a5a7-756f4c6bad1b"). InnerVolumeSpecName "kube-api-access-2gpf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.629625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.629493    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:13.629625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.629537    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9" pod "442db502-38c6-4dfa-b9e3-1e03b0461ee0" (UID: "442db502-38c6-4dfa-b9e3-1e03b0461ee0"). InnerVolumeSpecName "kube-api-access-t7hj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:13.724660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724627    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724657    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724668    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724677    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7hj9\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-kube-api-access-t7hj9\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724688    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724697    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724706    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724714    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724722    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724732    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724740    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724748    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724756    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724763    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724772    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724780    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724789    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64709aa4-be02-4056-a5a7-756f4c6bad1b-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724797    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvh6\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-kube-api-access-svvh6\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724806    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ed1963-4270-4551-a719-175e5ca80aa7-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724815    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724823    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724831    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64709aa4-be02-4056-a5a7-756f4c6bad1b-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724838    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ed1963-4270-4551-a719-175e5ca80aa7-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724848    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gpf6\" (UniqueName: \"kubernetes.io/projected/64709aa4-be02-4056-a5a7-756f4c6bad1b-kube-api-access-2gpf6\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.724965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.724856    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442db502-38c6-4dfa-b9e3-1e03b0461ee0-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:13.886695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.886656    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"]
Nov 03 03:31:13.887264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887248    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy"
Nov 03 03:31:13.887307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887267    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy"
Nov 03 03:31:13.887307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887280    5515 cpu_manager.go:414] "containerMap: removing container" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy"
Nov 03 03:31:13.887307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887284    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy"
Nov 03 03:31:13.887307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887300    5515 cpu_manager.go:414] "containerMap: removing container" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy"
Nov 03 03:31:13.887307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887305    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887318    5515 cpu_manager.go:414] "containerMap: removing container" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887323    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887331    5515 cpu_manager.go:414] "containerMap: removing container" podUID="93085299-7eb4-495f-ad03-f95594fc176a" containerName="payload-pre-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887336    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93085299-7eb4-495f-ad03-f95594fc176a" containerName="payload-pre-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887344    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887348    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887359    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887363    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887372    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887376    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887381    5515 cpu_manager.go:414] "containerMap: removing container" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887385    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887398    5515 cpu_manager.go:414] "containerMap: removing container" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887402    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887407    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1baaede1-f5ca-427b-a294-9718bf6481d4" containerName="payload-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887411    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baaede1-f5ca-427b-a294-9718bf6481d4" containerName="payload-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887421    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6b8d5e33-b51b-4707-8eda-88fa0881d9b6" containerName="payload-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887426    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8d5e33-b51b-4707-8eda-88fa0881d9b6" containerName="payload-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887434    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887438    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887453    5515 cpu_manager.go:414] "containerMap: removing container" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887457    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887467    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f61737ad-a4a3-4d6e-b163-99e6cbda21c7" containerName="payload-pre-processing"
Nov 03 03:31:13.887462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887471    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61737ad-a4a3-4d6e-b163-99e6cbda21c7" containerName="payload-pre-processing"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887591    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce378bb-7357-46ae-8159-065d109e2baa" containerName="cleanup"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887604    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887610    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baaede1-f5ca-427b-a294-9718bf6481d4" containerName="payload-processing"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887616    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8d5e33-b51b-4707-8eda-88fa0881d9b6" containerName="payload-processing"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887623    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887631    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb39043a-38b8-421b-825b-240ffbd44ec4" containerName="cleanup"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887637    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61737ad-a4a3-4d6e-b163-99e6cbda21c7" containerName="payload-pre-processing"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887645    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887650    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887658    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="45285d6d-5238-4608-ad26-49df796027b2" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887665    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887671    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="93085299-7eb4-495f-ad03-f95594fc176a" containerName="payload-pre-processing"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887677    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ac0ec-d6a4-4e93-ab05-787de1e79f0a" containerName="cleanup"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887684    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" containerName="istio-proxy"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887689    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4df851-36f7-4c6d-8cb2-a4a6d89de81a" containerName="cleanup"
Nov 03 03:31:13.888233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.887694    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc36b093-09b4-43f3-88c4-294c70a0dbb5" containerName="cleanup"
Nov 03 03:31:13.892638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.892619    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:13.969229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.958802    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-687adc-openshift-default-dockercfg-bbqbq\""
Nov 03 03:31:13.971615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.971576    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"]
Nov 03 03:31:13.986787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:13.986755    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:14.028857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.028823    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.028857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.028861    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.028883    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.029022    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.029172    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.029250    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdmv7\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.029282    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.029443    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.029788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.029771    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131110    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131162    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131198    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131246    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131359    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131439    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdmv7\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131466    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131507    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.131786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.131618    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.132503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.132475    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.151642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.144087    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.151642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.144307    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.151642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.144838    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.151642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.145259    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.151642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.147279    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:14.198594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.193953    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.233575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.227312    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.263598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.260801    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdmv7\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.282594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.281653    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t" event={"ID":"64709aa4-be02-4056-a5a7-756f4c6bad1b","Type":"ContainerDied","Data":"351a0c579e588ad0d998273c9fea8c83dfc10e7acd81e0716fe5ff43f2207c10"}
Nov 03 03:31:14.282594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.281724    5515 scope.go:122] "RemoveContainer" containerID="fe69845a8ac5286473ce1b281b955ffef0117d44a1817919e5d81a52c2c4e22e"
Nov 03 03:31:14.282594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.281911    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"
Nov 03 03:31:14.299608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.298106    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token\") pod \"e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") " pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.329585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.320159    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:14.358587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.354583    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl" event={"ID":"6388c0bd-7d55-4dd5-b203-ab82a6102a25","Type":"ContainerDied","Data":"9a89e13fb49932f5266b399467716218afc0535709f2cfd49f090b62b903ff53"}
Nov 03 03:31:14.358587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.354731    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"
Nov 03 03:31:14.387860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.370750    5515 generic.go:356] "Generic (PLEG): container finished" podID="e7ed1963-4270-4551-a719-175e5ca80aa7" containerID="a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977" exitCode=0
Nov 03 03:31:14.387860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.370959    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" event={"ID":"e7ed1963-4270-4551-a719-175e5ca80aa7","Type":"ContainerDied","Data":"a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977"}
Nov 03 03:31:14.387860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.371013    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z" event={"ID":"e7ed1963-4270-4551-a719-175e5ca80aa7","Type":"ContainerDied","Data":"594bbf825e2f76e5cfbc86306e25249e9d2b9df71fd2c57a4125d358d1815b30"}
Nov 03 03:31:14.387860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.371116    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"
Nov 03 03:31:14.387860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.383688    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g" event={"ID":"442db502-38c6-4dfa-b9e3-1e03b0461ee0","Type":"ContainerDied","Data":"4dcf51a82ebe8718f198884b0cb478e5687dbc3b9c51bf44aac9f222d171db4d"}
Nov 03 03:31:14.387860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.383828    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"
Nov 03 03:31:14.426401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.425054    5515 scope.go:122] "RemoveContainer" containerID="ec88d68992a90524c670f18793d1ada4eb8b7e71d2a35b6573ef3d40bbb58476"
Nov 03 03:31:14.457863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.451365    5515 scope.go:122] "RemoveContainer" containerID="a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977"
Nov 03 03:31:14.471219 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.470979    5515 scope.go:122] "RemoveContainer" containerID="a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977"
Nov 03 03:31:14.471669 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:14.471318    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977\": container with ID starting with a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977 not found: ID does not exist" containerID="a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977"
Nov 03 03:31:14.471669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.471343    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977"} err="failed to get container status \"a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977\": rpc error: code = NotFound desc = could not find container \"a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977\": container with ID starting with a084c612b12c4a72dd49c2b30bfb6a1a326a4d2b68dc17dfc2d08dd17c749977 not found: ID does not exist"
Nov 03 03:31:14.471669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.471361    5515 scope.go:122] "RemoveContainer" containerID="7e79f283a52fdb436935e49eb595d023e16b76035b04319fafaaf8b1d8cc1062"
Nov 03 03:31:14.506679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.506652    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:14.650654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.650623    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"]
Nov 03 03:31:14.655035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.655007    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-6vwpl"]
Nov 03 03:31:14.765192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.765106    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"]
Nov 03 03:31:14.771459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.771428    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-sn46t"]
Nov 03 03:31:14.883028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.882973    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"]
Nov 03 03:31:14.886281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.886254    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-k4l7g"]
Nov 03 03:31:14.895350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.895317    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442db502-38c6-4dfa-b9e3-1e03b0461ee0" path="/var/lib/kubelet/pods/442db502-38c6-4dfa-b9e3-1e03b0461ee0/volumes"
Nov 03 03:31:14.895919 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:14.895897    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfe2231_d5e3_4ed6_88c9_07483aae2c43.slice/crio-78d4ea80778cc9e5615cb956ed024182af4169939baa710e930e24304f32328c WatchSource:0}: Error finding container 78d4ea80778cc9e5615cb956ed024182af4169939baa710e930e24304f32328c: Status 404 returned error can't find the container with id 78d4ea80778cc9e5615cb956ed024182af4169939baa710e930e24304f32328c
Nov 03 03:31:14.896034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.895960    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6388c0bd-7d55-4dd5-b203-ab82a6102a25" path="/var/lib/kubelet/pods/6388c0bd-7d55-4dd5-b203-ab82a6102a25/volumes"
Nov 03 03:31:14.896561 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.896526    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64709aa4-be02-4056-a5a7-756f4c6bad1b" path="/var/lib/kubelet/pods/64709aa4-be02-4056-a5a7-756f4c6bad1b/volumes"
Nov 03 03:31:14.897233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.897213    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"]
Nov 03 03:31:14.898247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.898219    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:31:14.898319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.898277    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:31:14.898319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.898306    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:31:14.997433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:14.997395    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"]
Nov 03 03:31:15.002053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:15.002025    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-68b670-openshift-default-6848c559df-76h2z"]
Nov 03 03:31:15.392994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:15.392960    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" event={"ID":"3bfe2231-d5e3-4ed6-88c9-07483aae2c43","Type":"ContainerStarted","Data":"3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"}
Nov 03 03:31:15.393185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:15.393010    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" event={"ID":"3bfe2231-d5e3-4ed6-88c9-07483aae2c43","Type":"ContainerStarted","Data":"78d4ea80778cc9e5615cb956ed024182af4169939baa710e930e24304f32328c"}
Nov 03 03:31:15.507275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:15.507239    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:15.512861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:15.512835    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:15.515427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:15.515378    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podStartSLOduration=2.5153623769999998 podStartE2EDuration="2.515362377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:15.514917822 +0000 UTC m=+1766.745810243" watchObservedRunningTime="2026-11-03 03:31:15.515362377 +0000 UTC m=+1766.746254800"
Nov 03 03:31:16.402565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:16.402530    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:16.403516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:16.403499    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:31:16.782327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:16.782245    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:16.896097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:16.896062    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ed1963-4270-4551-a719-175e5ca80aa7" path="/var/lib/kubelet/pods/e7ed1963-4270-4551-a719-175e5ca80aa7/volumes"
Nov 03 03:31:16.958288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:16.958258    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:19.108464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.108426    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:19.308736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.308703    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:19.508617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.508497    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:19.692577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.686404    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:19.712716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.697377    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"]
Nov 03 03:31:19.712716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.698606    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a614825-5146-44d1-8753-a72927a9e298" containerName="istio-proxy"
Nov 03 03:31:19.729576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.728359    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:19.806629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.806458    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9902a0c2\""
Nov 03 03:31:19.809585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.807849    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9902a0c2-dockercfg-npd95\""
Nov 03 03:31:19.834578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.818259    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"]
Nov 03 03:31:19.887612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.878149    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"]
Nov 03 03:31:19.887612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.880757    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:19.887612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.880889    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8jm\" (UniqueName: \"kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:19.934528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.934020    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:19.950120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.950093    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"]
Nov 03 03:31:19.982395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.982357    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:19.982605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.982515    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8jm\" (UniqueName: \"kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:19.983115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:19.983092    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:20.044904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.044820    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8jm\" (UniqueName: \"kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm\") pod \"payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") " pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:20.084613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.084528    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.084863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.084843    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtd2\" (UniqueName: \"kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.137169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.137134    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:20.186869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.186828    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtd2\" (UniqueName: \"kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.187037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.187011    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.187887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.187860    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.249444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.249408    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtd2\" (UniqueName: \"kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2\") pod \"payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.259200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.259168    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:20.694041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.694013    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"]
Nov 03 03:31:20.694789 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:20.694761    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87980c56_7904_4dcb_8297_36cfd18adc3a.slice/crio-1ea49ee046b6076765148f2b615c8d11492560270949be4baf24a81eb125b6c1 WatchSource:0}: Error finding container 1ea49ee046b6076765148f2b615c8d11492560270949be4baf24a81eb125b6c1: Status 404 returned error can't find the container with id 1ea49ee046b6076765148f2b615c8d11492560270949be4baf24a81eb125b6c1
Nov 03 03:31:20.733660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.733632    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:31:20.743855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:20.743831    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"]
Nov 03 03:31:20.746603 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:20.746578    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce923f5_be32_4b99_bcf7_1f0084f8c81a.slice/crio-a261edd17da4332acf1e6148ee82d2d7587bf75b02811dca70cb5f611712f098 WatchSource:0}: Error finding container a261edd17da4332acf1e6148ee82d2d7587bf75b02811dca70cb5f611712f098: Status 404 returned error can't find the container with id a261edd17da4332acf1e6148ee82d2d7587bf75b02811dca70cb5f611712f098
Nov 03 03:31:20.985716 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:20.985630    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:31:21.706939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.705325    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz" event={"ID":"0ce923f5-be32-4b99-bcf7-1f0084f8c81a","Type":"ContainerStarted","Data":"4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419"}
Nov 03 03:31:21.706939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.705442    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz" event={"ID":"0ce923f5-be32-4b99-bcf7-1f0084f8c81a","Type":"ContainerStarted","Data":"a261edd17da4332acf1e6148ee82d2d7587bf75b02811dca70cb5f611712f098"}
Nov 03 03:31:21.706939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.706649    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:21.717574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.710089    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l" event={"ID":"87980c56-7904-4dcb-8297-36cfd18adc3a","Type":"ContainerStarted","Data":"7d119f6fa63746454a10171fe2031ddef843a2413a289ebdc2b4ef379e25dc58"}
Nov 03 03:31:21.717574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.710137    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l" event={"ID":"87980c56-7904-4dcb-8297-36cfd18adc3a","Type":"ContainerStarted","Data":"1ea49ee046b6076765148f2b615c8d11492560270949be4baf24a81eb125b6c1"}
Nov 03 03:31:21.717574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.710862    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:21.841283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:21.841219    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz" podStartSLOduration=2.841200796 podStartE2EDuration="2.841200796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:21.840906901 +0000 UTC m=+1773.071799318" watchObservedRunningTime="2026-11-03 03:31:21.841200796 +0000 UTC m=+1773.072093217"
Nov 03 03:31:23.996573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:23.994146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:24.545895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:24.545855    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:24.821516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:24.821421    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:25.122582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.122002    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l" podStartSLOduration=6.121981017 podStartE2EDuration="6.121981017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:22.001972421 +0000 UTC m=+1773.232864840" watchObservedRunningTime="2026-11-03 03:31:25.121981017 +0000 UTC m=+1776.352873439"
Nov 03 03:31:25.133573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.131660    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"]
Nov 03 03:31:25.146859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.146828    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.198243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.192911    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"]
Nov 03 03:31:25.395655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.389352    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-687adc-dockercfg-hgbbg\""
Nov 03 03:31:25.395655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.389436    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-687adc\""
Nov 03 03:31:25.395655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.390942    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume\") pod \"payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") " pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.395655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.391138    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptswg\" (UniqueName: \"kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg\") pod \"payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") " pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.397636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.397612    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"]
Nov 03 03:31:25.439932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.413934    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.496922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.496880    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj8t\" (UniqueName: \"kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t\") pod \"payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.497135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.497114    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume\") pod \"payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") " pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.497271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.497258    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.497387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.497377    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptswg\" (UniqueName: \"kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg\") pod \"payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") " pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.498717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.498697    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume\") pod \"payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") " pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.516151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.516115    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"]
Nov 03 03:31:25.563831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.563796    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptswg\" (UniqueName: \"kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg\") pod \"payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") " pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:25.598625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.598593    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj8t\" (UniqueName: \"kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t\") pod \"payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.598787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.598688    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.599336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.599318    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.663114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.663040    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj8t\" (UniqueName: \"kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t\") pod \"payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.766264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.766234    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:25.795807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.795782    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:25.828671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:25.827465    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:26.278997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.278970    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"]
Nov 03 03:31:26.280011 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:26.279983    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd80adf_3acf_4cf7_805f_777bdbc84bb6.slice/crio-d77e36520bd746897cfe30aaa907f8900449ee9aea7a2fcbecefa013f2b50c6b WatchSource:0}: Error finding container d77e36520bd746897cfe30aaa907f8900449ee9aea7a2fcbecefa013f2b50c6b: Status 404 returned error can't find the container with id d77e36520bd746897cfe30aaa907f8900449ee9aea7a2fcbecefa013f2b50c6b
Nov 03 03:31:26.333674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.333649    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"]
Nov 03 03:31:26.334069 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:26.334039    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1a6c1f_9790_4dfd_b1d0_ddddcbc66018.slice/crio-1675b513ed9580a107a86211039f5f7b140a2cb8892142df4ca236e0418d941c WatchSource:0}: Error finding container 1675b513ed9580a107a86211039f5f7b140a2cb8892142df4ca236e0418d941c: Status 404 returned error can't find the container with id 1675b513ed9580a107a86211039f5f7b140a2cb8892142df4ca236e0418d941c
Nov 03 03:31:26.884826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.884795    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq" event={"ID":"ccd80adf-3acf-4cf7-805f-777bdbc84bb6","Type":"ContainerStarted","Data":"70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688"}
Nov 03 03:31:26.885030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.884844    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq" event={"ID":"ccd80adf-3acf-4cf7-805f-777bdbc84bb6","Type":"ContainerStarted","Data":"d77e36520bd746897cfe30aaa907f8900449ee9aea7a2fcbecefa013f2b50c6b"}
Nov 03 03:31:26.885030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.884868    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:26.901453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.901427    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:26.901632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.901474    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8" event={"ID":"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018","Type":"ContainerStarted","Data":"164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77"}
Nov 03 03:31:26.901632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:26.901495    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8" event={"ID":"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018","Type":"ContainerStarted","Data":"1675b513ed9580a107a86211039f5f7b140a2cb8892142df4ca236e0418d941c"}
Nov 03 03:31:27.005183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:27.005132    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq" podStartSLOduration=2.00511753 podStartE2EDuration="2.00511753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:27.002812336 +0000 UTC m=+1778.233704755" watchObservedRunningTime="2026-11-03 03:31:27.00511753 +0000 UTC m=+1778.236009949"
Nov 03 03:31:27.134572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:27.133843    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8" podStartSLOduration=2.133824683 podStartE2EDuration="2.133824683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:27.132383672 +0000 UTC m=+1778.363276098" watchObservedRunningTime="2026-11-03 03:31:27.133824683 +0000 UTC m=+1778.364717169"
Nov 03 03:31:27.763735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:27.763699    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:28.506404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:28.506368    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:29.014865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.014830    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"]
Nov 03 03:31:29.019196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.019177    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066611    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066675    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066717    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9tm\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066817    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066868    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066896    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.066960    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.067015    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.073573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.067150    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.081578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.080930    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-211ed3-openshift-default-dockercfg-wgmb7\""
Nov 03 03:31:29.124578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.117574    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.181810    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.181857    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.181928    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.181980    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.182105    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.182135    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.182170    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.182206    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9tm\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.182576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.182339    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.215656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.203626    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.215656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.204310    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.215656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.204566    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.215656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.204955    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.236063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.218296    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.282635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.272934    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.282635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.275855    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.315577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.287922    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"]
Nov 03 03:31:29.358573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.351247    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:29.380634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.365947    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.404572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.397667    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9tm\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm\") pod \"e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") " pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:29.634169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:29.634133    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:30.072008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:30.071984    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"]
Nov 03 03:31:30.072522 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:30.072497    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb11c27_bf88_4249_8603_f41209cc645b.slice/crio-e4e62258c3b463941ced565198e62146c6b96e43aaf3fc251746d7da496d068e WatchSource:0}: Error finding container e4e62258c3b463941ced565198e62146c6b96e43aaf3fc251746d7da496d068e: Status 404 returned error can't find the container with id e4e62258c3b463941ced565198e62146c6b96e43aaf3fc251746d7da496d068e
Nov 03 03:31:30.075268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:30.075236    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:31:30.075337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:30.075304    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:31:30.075337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:30.075331    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:31:30.935045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:30.935006    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" event={"ID":"7cb11c27-bf88-4249-8603-f41209cc645b","Type":"ContainerStarted","Data":"f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"}
Nov 03 03:31:30.935217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:30.935099    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" event={"ID":"7cb11c27-bf88-4249-8603-f41209cc645b","Type":"ContainerStarted","Data":"e4e62258c3b463941ced565198e62146c6b96e43aaf3fc251746d7da496d068e"}
Nov 03 03:31:31.079670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:31.079634    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:31.110967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:31.110909    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podStartSLOduration=3.110894748 podStartE2EDuration="3.110894748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:31.107755094 +0000 UTC m=+1782.338647514" watchObservedRunningTime="2026-11-03 03:31:31.110894748 +0000 UTC m=+1782.341787166"
Nov 03 03:31:31.634396 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:31.634362    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:31.640063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:31.640025    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:31.941765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:31.941675    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:31.942841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:31.942819    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:31:33.731276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:33.731245    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:31:33.731712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:33.731319    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:31:34.094188 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:34.093011    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:34.277856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:34.277822    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:34.464504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:34.464473    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:35.076039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:35.076006    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:35.233010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:35.232979    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:36.399832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:36.399804    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:37.224706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.224675    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"]
Nov 03 03:31:37.224883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.224860    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy" containerID="cri-o://1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1" gracePeriod=30
Nov 03 03:31:37.274918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.274887    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"]
Nov 03 03:31:37.275174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.275136    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" containerID="cri-o://0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be" gracePeriod=30
Nov 03 03:31:37.277792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.277765    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"]
Nov 03 03:31:37.277971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.277948    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy" containerID="cri-o://f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7" gracePeriod=30
Nov 03 03:31:37.326456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.326421    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"]
Nov 03 03:31:37.326654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.326633    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy" containerID="cri-o://e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b" gracePeriod=30
Nov 03 03:31:37.362298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.362246    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:37.640797 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.640757    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:37.791619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.791581    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:37.844902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.844873    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"]
Nov 03 03:31:37.849218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.849199    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:37.926101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.919253    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-211ed3-dockercfg-f7zcd\""
Nov 03 03:31:37.926101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.919371    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-211ed3\""
Nov 03 03:31:37.942844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.941078    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:37.942844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.941338    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbm6\" (UniqueName: \"kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6\") pod \"payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:37.976576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:37.952406    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"]
Nov 03 03:31:38.140032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.136477    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:31:38.159210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.159184    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:31:38.197905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.197825    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"]
Nov 03 03:31:38.219006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.218969    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:38.219383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.219365    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbm6\" (UniqueName: \"kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6\") pod \"payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:38.219989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.219926    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:38.220858 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.220834    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:38.226248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.225922    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"]
Nov 03 03:31:38.226248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.226038    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.287424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.287394    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbm6\" (UniqueName: \"kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6\") pod \"payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:38.321132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.321104    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume\") pod \"payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") " pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.321501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.321474    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tfr\" (UniqueName: \"kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr\") pod \"payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") " pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.439586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.436099    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tfr\" (UniqueName: \"kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr\") pod \"payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") " pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.439586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.436362    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume\") pod \"payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") " pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.447574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.445677    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume\") pod \"payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") " pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.471632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.463918    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:38.577768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.561456    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tfr\" (UniqueName: \"kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr\") pod \"payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") " pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:38.846258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:38.846225    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:39.028500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:39.028406    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:39.044854 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:39.044823    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"]
Nov 03 03:31:39.045135 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:39.045112    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd91b6d_039a_4101_a8ec_48804f4c1cd2.slice/crio-fc1c43a7786afdc9a36e54b9e498498e4f21ad01bdb9d783d628808baae74feb WatchSource:0}: Error finding container fc1c43a7786afdc9a36e54b9e498498e4f21ad01bdb9d783d628808baae74feb: Status 404 returned error can't find the container with id fc1c43a7786afdc9a36e54b9e498498e4f21ad01bdb9d783d628808baae74feb
Nov 03 03:31:39.112458 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:39.112429    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8" event={"ID":"6fd91b6d-039a-4101-a8ec-48804f4c1cd2","Type":"ContainerStarted","Data":"fc1c43a7786afdc9a36e54b9e498498e4f21ad01bdb9d783d628808baae74feb"}
Nov 03 03:31:39.224968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:39.224940    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"]
Nov 03 03:31:39.225749 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:39.225723    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9188f72b_439a_4dd8_ad15_06099a4f5614.slice/crio-32749fe08bad7f75f680aefb34f2a99eaf3298da71f2853fa1788963c49359de WatchSource:0}: Error finding container 32749fe08bad7f75f680aefb34f2a99eaf3298da71f2853fa1788963c49359de: Status 404 returned error can't find the container with id 32749fe08bad7f75f680aefb34f2a99eaf3298da71f2853fa1788963c49359de
Nov 03 03:31:39.418513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:39.418477    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:31:39.418738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:39.418535    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:31:40.121988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.121955    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl" event={"ID":"9188f72b-439a-4dd8-ad15-06099a4f5614","Type":"ContainerStarted","Data":"47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31"}
Nov 03 03:31:40.122456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.122007    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl" event={"ID":"9188f72b-439a-4dd8-ad15-06099a4f5614","Type":"ContainerStarted","Data":"32749fe08bad7f75f680aefb34f2a99eaf3298da71f2853fa1788963c49359de"}
Nov 03 03:31:40.122456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.122044    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:40.123762 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.123735    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8" event={"ID":"6fd91b6d-039a-4101-a8ec-48804f4c1cd2","Type":"ContainerStarted","Data":"74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1"}
Nov 03 03:31:40.123897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.123786    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:40.241456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.241407    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl" podStartSLOduration=3.241394416 podStartE2EDuration="3.241394416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:40.239915797 +0000 UTC m=+1791.470808277" watchObservedRunningTime="2026-11-03 03:31:40.241394416 +0000 UTC m=+1791.472286832"
Nov 03 03:31:40.368579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:40.362743    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8" podStartSLOduration=3.362720424 podStartE2EDuration="3.362720424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:31:40.359981651 +0000 UTC m=+1791.590874073" watchObservedRunningTime="2026-11-03 03:31:40.362720424 +0000 UTC m=+1791.593612841"
Nov 03 03:31:41.720864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:41.720833    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:42.569025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.569001    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:31:42.660995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.660965    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.676753    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.676809    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.676967    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677089    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677129    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677213    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677203    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677354    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677384    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.677573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.677422    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token\") pod \"095440b1-e9d8-4acc-90c5-d973820d82d2\" (UID: \"095440b1-e9d8-4acc-90c5-d973820d82d2\") "
Nov 03 03:31:42.679134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.678846    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/095440b1-e9d8-4acc-90c5-d973820d82d2-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.680319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.679817    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.680748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.680525    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.680956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.680910    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.681374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.681354    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.682997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.682975    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:42.683253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.683216    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "kube-api-access-8xjjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.684891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.684866    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.685641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.685617    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy" pod "095440b1-e9d8-4acc-90c5-d973820d82d2" (UID: "095440b1-e9d8-4acc-90c5-d973820d82d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.703857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.703832    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:31:42.716620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.716596    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:31:42.780279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780183    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780267    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780320    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780351    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780395    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780445    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780527    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780609    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780679    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780754    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780797    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.780817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780798    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.781256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780856    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.781256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.780944    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.781256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781092    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.781256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781229    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.781256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781240    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.781559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781303    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.781559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781341    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.781559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781378    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token\") pod \"5995f7c2-777d-4518-ab7e-712a54ca57e4\" (UID: \"5995f7c2-777d-4518-ab7e-712a54ca57e4\") "
Nov 03 03:31:42.781559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781414    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.781559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.781466    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo\") pod \"71bb2648-08d7-436b-82fa-8a06538a04ae\" (UID: \"71bb2648-08d7-436b-82fa-8a06538a04ae\") "
Nov 03 03:31:42.782459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782431    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782456    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782464    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjjd\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-kube-api-access-8xjjd\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782497    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782512    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782525    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782538    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782573    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.782599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782586    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.783017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782730    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.783017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782769    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.783017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782796    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.783017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782800    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.783017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782811    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/095440b1-e9d8-4acc-90c5-d973820d82d2-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.783017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.782841    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:42.783594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.783177    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.783594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.783352    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.783594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.783564    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.784786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.784757    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:42.785746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.785722    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "kube-api-access-8hvwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.785880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.785759    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.786048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.785931    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "kube-api-access-gwsnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.786283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.786259    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy" pod "5995f7c2-777d-4518-ab7e-712a54ca57e4" (UID: "5995f7c2-777d-4518-ab7e-712a54ca57e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.786588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.786528    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.786998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.786965    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.787428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.787403    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo" pod "71bb2648-08d7-436b-82fa-8a06538a04ae" (UID: "71bb2648-08d7-436b-82fa-8a06538a04ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:42.883665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.883634    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.883859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.883701    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.883859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.883746    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.883859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.883821    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.884054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.883897    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.884054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.883954    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.884054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884004    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.884232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884135    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.884232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884150    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:42.884232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884187    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo\") pod \"256bd96e-dc29-439c-9978-a0cc51c869c7\" (UID: \"256bd96e-dc29-439c-9978-a0cc51c869c7\") "
Nov 03 03:31:42.884405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884359    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.884405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884389    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.884639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884611    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.884822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884801    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.885036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.884994    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb2648-08d7-436b-82fa-8a06538a04ae-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885014    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/256bd96e-dc29-439c-9978-a0cc51c869c7-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885030    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885044    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885058    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwsnl\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-kube-api-access-gwsnl\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885074    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885087    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885102    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5995f7c2-777d-4518-ab7e-712a54ca57e4-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885113    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885126    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885139    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885150    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb2648-08d7-436b-82fa-8a06538a04ae-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885165    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885177    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885189    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvwx\" (UniqueName: \"kubernetes.io/projected/71bb2648-08d7-436b-82fa-8a06538a04ae-kube-api-access-8hvwx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885200    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb2648-08d7-436b-82fa-8a06538a04ae-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885213    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885224    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.885349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.885237    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5995f7c2-777d-4518-ab7e-712a54ca57e4-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.886929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.886905    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:42.887182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.887152    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "kube-api-access-lljnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.887833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.887810    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:42.888105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.888078    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo" pod "256bd96e-dc29-439c-9978-a0cc51c869c7" (UID: "256bd96e-dc29-439c-9978-a0cc51c869c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:42.987036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.986996    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.987036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.987026    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.987036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.987043    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lljnj\" (UniqueName: \"kubernetes.io/projected/256bd96e-dc29-439c-9978-a0cc51c869c7-kube-api-access-lljnj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:42.987284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:42.987058    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/256bd96e-dc29-439c-9978-a0cc51c869c7-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:43.147086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.147051    5515 generic.go:356] "Generic (PLEG): container finished" podID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerID="e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b" exitCode=0
Nov 03 03:31:43.147301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.147276    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"
Nov 03 03:31:43.148129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.148101    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" event={"ID":"256bd96e-dc29-439c-9978-a0cc51c869c7","Type":"ContainerDied","Data":"e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b"}
Nov 03 03:31:43.148269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.148161    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg" event={"ID":"256bd96e-dc29-439c-9978-a0cc51c869c7","Type":"ContainerDied","Data":"08de1684cfe2db3e960e2f46b6ebcd9734f100ad81366cd889610d897b624ade"}
Nov 03 03:31:43.148269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.148192    5515 scope.go:122] "RemoveContainer" containerID="e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b"
Nov 03 03:31:43.151916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.151889    5515 generic.go:356] "Generic (PLEG): container finished" podID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerID="0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be" exitCode=0
Nov 03 03:31:43.152013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.151956    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"
Nov 03 03:31:43.152013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.151997    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" event={"ID":"5995f7c2-777d-4518-ab7e-712a54ca57e4","Type":"ContainerDied","Data":"0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be"}
Nov 03 03:31:43.152134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.152036    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd" event={"ID":"5995f7c2-777d-4518-ab7e-712a54ca57e4","Type":"ContainerDied","Data":"4e0ef5218ff12be2f15302952dbaed3faba98e3b9710f9de1c6d2ebb665b8b85"}
Nov 03 03:31:43.153878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.153855    5515 generic.go:356] "Generic (PLEG): container finished" podID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerID="1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1" exitCode=0
Nov 03 03:31:43.153979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.153907    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"
Nov 03 03:31:43.153979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.153939    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" event={"ID":"095440b1-e9d8-4acc-90c5-d973820d82d2","Type":"ContainerDied","Data":"1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1"}
Nov 03 03:31:43.154082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.153991    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd" event={"ID":"095440b1-e9d8-4acc-90c5-d973820d82d2","Type":"ContainerDied","Data":"1d450f2c9c2000fd7fd72a2b7905dafa9eee56bd8bd2d6adbcafc2024660a410"}
Nov 03 03:31:43.157098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.157077    5515 generic.go:356] "Generic (PLEG): container finished" podID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerID="f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7" exitCode=0
Nov 03 03:31:43.157215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.157119    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"
Nov 03 03:31:43.157215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.157132    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" event={"ID":"71bb2648-08d7-436b-82fa-8a06538a04ae","Type":"ContainerDied","Data":"f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7"}
Nov 03 03:31:43.157215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.157184    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2" event={"ID":"71bb2648-08d7-436b-82fa-8a06538a04ae","Type":"ContainerDied","Data":"1d33c8b0a78fd1863be58c280f030377248f76dda217d52999c9ff1c350b3123"}
Nov 03 03:31:43.165595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.165573    5515 scope.go:122] "RemoveContainer" containerID="e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b"
Nov 03 03:31:43.165943 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:43.165919    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b\": container with ID starting with e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b not found: ID does not exist" containerID="e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b"
Nov 03 03:31:43.166051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.165957    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b"} err="failed to get container status \"e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b\": rpc error: code = NotFound desc = could not find container \"e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b\": container with ID starting with e81b41915248911c5db95bf78f69b8bb237f80ee0dfb035e5e37df0af2a9cf4b not found: ID does not exist"
Nov 03 03:31:43.166051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.165979    5515 scope.go:122] "RemoveContainer" containerID="0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be"
Nov 03 03:31:43.179018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.178992    5515 scope.go:122] "RemoveContainer" containerID="0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be"
Nov 03 03:31:43.179366 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:43.179343    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be\": container with ID starting with 0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be not found: ID does not exist" containerID="0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be"
Nov 03 03:31:43.179474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.179375    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be"} err="failed to get container status \"0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be\": rpc error: code = NotFound desc = could not find container \"0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be\": container with ID starting with 0794c63dabc9c0d2315aa954fd1a72d54d9d6a12c7813df5b0cbe929931710be not found: ID does not exist"
Nov 03 03:31:43.179474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.179397    5515 scope.go:122] "RemoveContainer" containerID="1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1"
Nov 03 03:31:43.191703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.191680    5515 scope.go:122] "RemoveContainer" containerID="1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1"
Nov 03 03:31:43.192035 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:43.192015    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1\": container with ID starting with 1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1 not found: ID does not exist" containerID="1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1"
Nov 03 03:31:43.192113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.192043    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1"} err="failed to get container status \"1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1\": rpc error: code = NotFound desc = could not find container \"1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1\": container with ID starting with 1f806aa08baeaa506d1ccbc8506247502427eebc8b8fb8e613f0ce64608a07c1 not found: ID does not exist"
Nov 03 03:31:43.192113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.192070    5515 scope.go:122] "RemoveContainer" containerID="f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7"
Nov 03 03:31:43.202510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.202479    5515 scope.go:122] "RemoveContainer" containerID="f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7"
Nov 03 03:31:43.202804 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:43.202775    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7\": container with ID starting with f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7 not found: ID does not exist" containerID="f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7"
Nov 03 03:31:43.202874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.202801    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7"} err="failed to get container status \"f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7\": rpc error: code = NotFound desc = could not find container \"f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7\": container with ID starting with f56d3bf9660aacd4599a68607586bc4c8c8dbc78f48467f30195de7de7a4cca7 not found: ID does not exist"
Nov 03 03:31:43.318177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.318135    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"]
Nov 03 03:31:43.321698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.321674    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-9lwh2"]
Nov 03 03:31:43.433287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.433212    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"]
Nov 03 03:31:43.439426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.439403    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-zcpjg"]
Nov 03 03:31:43.551915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.551562    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"]
Nov 03 03:31:43.554678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.554658    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-hdzgd"]
Nov 03 03:31:43.666748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.666713    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"]
Nov 03 03:31:43.669844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:43.669825    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-8rrpd"]
Nov 03 03:31:44.895624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:44.895591    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" path="/var/lib/kubelet/pods/095440b1-e9d8-4acc-90c5-d973820d82d2/volumes"
Nov 03 03:31:44.896443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:44.896420    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" path="/var/lib/kubelet/pods/256bd96e-dc29-439c-9978-a0cc51c869c7/volumes"
Nov 03 03:31:44.896968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:44.896947    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" path="/var/lib/kubelet/pods/5995f7c2-777d-4518-ab7e-712a54ca57e4/volumes"
Nov 03 03:31:44.897437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:44.897419    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" path="/var/lib/kubelet/pods/71bb2648-08d7-436b-82fa-8a06538a04ae/volumes"
Nov 03 03:31:45.144079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:45.144037    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"]
Nov 03 03:31:45.144309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:45.144284    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" containerID="cri-o://3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a" gracePeriod=30
Nov 03 03:31:46.159516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:46.159480    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"]
Nov 03 03:31:46.288873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:46.288841    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:46.435433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:46.435353    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:46.592780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:46.592734    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:47.021680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:47.021645    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:49.761680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:49.761646    5515 scope.go:122] "RemoveContainer" containerID="ec9bf85bbfc23df7f1fdd025046f4dd908017ea182c4ac036165c9c1f2a2719b"
Nov 03 03:31:49.969281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:49.969259    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:31:49.972435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:49.972415    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:31:49.973226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:49.973206    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:31:49.977301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:49.977279    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:31:50.127652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.127622    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:31:50.238277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.238243    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 03:31:50.238454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.238303    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 03:31:50.418126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.418103    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:31:50.428334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428308    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428350    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428377    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428395    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428420    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428459    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428521    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428630    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428649    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:50.428833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428698    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7\") pod \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\" (UID: \"b7102b72-a704-44f6-a6a4-dbb727d47ed1\") "
Nov 03 03:31:50.428833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428719    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:50.429002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428843    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:50.429002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428861    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:50.429002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.428892    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:50.429271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.429248    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.429271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.429269    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.429648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.429279    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.429648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.429288    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.429648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.429295    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.431034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.431004    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:50.431034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.431020    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "kube-api-access-p8cz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:50.431203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.431185    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:31:50.431239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.431195    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy" pod "b7102b72-a704-44f6-a6a4-dbb727d47ed1" (UID: "b7102b72-a704-44f6-a6a4-dbb727d47ed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:31:50.530774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.530742    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8cz7\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-kube-api-access-p8cz7\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.530774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.530772    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.530996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.530783    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:50.530996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:50.530793    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7102b72-a704-44f6-a6a4-dbb727d47ed1-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:51.027367 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:51.027320    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:31:51.136425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.136373    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:31:51.137799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.137774    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:31:51.160771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.160722    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm" podUID="b6527b2f-0450-4b50-83ed-19262fd9671f" containerName="payload-pre-processing" containerID="cri-o://9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246" gracePeriod=25
Nov 03 03:31:51.223925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.223894    5515 generic.go:356] "Generic (PLEG): container finished" podID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerID="3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a" exitCode=0
Nov 03 03:31:51.224107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.223969    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"
Nov 03 03:31:51.224107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.223984    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" event={"ID":"b7102b72-a704-44f6-a6a4-dbb727d47ed1","Type":"ContainerDied","Data":"3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a"}
Nov 03 03:31:51.224107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.224064    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt" event={"ID":"b7102b72-a704-44f6-a6a4-dbb727d47ed1","Type":"ContainerDied","Data":"9ea59ca60eefdfc12e91c3b1c046b01f04c9286f9f8924f0469bbe7b4576ad58"}
Nov 03 03:31:51.224107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.224098    5515 scope.go:122] "RemoveContainer" containerID="3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a"
Nov 03 03:31:51.285531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.285241    5515 scope.go:122] "RemoveContainer" containerID="3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a"
Nov 03 03:31:51.288391 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:51.285959    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a\": container with ID starting with 3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a not found: ID does not exist" containerID="3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a"
Nov 03 03:31:51.288391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.286017    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a"} err="failed to get container status \"3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a\": rpc error: code = NotFound desc = could not find container \"3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a\": container with ID starting with 3b448ce6b37aa357ad90c43308b320d8d42f056b69efdb47d9e62f840092a13a not found: ID does not exist"
Nov 03 03:31:51.389305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.380642    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"]
Nov 03 03:31:51.434599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.434530    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-5fbkt"]
Nov 03 03:31:51.557695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.557623    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:31:51.592604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.592563    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume\") pod \"b6527b2f-0450-4b50-83ed-19262fd9671f\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") "
Nov 03 03:31:51.593366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.592880    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85\" (UniqueName: \"kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85\") pod \"b6527b2f-0450-4b50-83ed-19262fd9671f\" (UID: \"b6527b2f-0450-4b50-83ed-19262fd9671f\") "
Nov 03 03:31:51.600936 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.598078    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume" pod "b6527b2f-0450-4b50-83ed-19262fd9671f" (UID: "b6527b2f-0450-4b50-83ed-19262fd9671f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:31:51.600936 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.600881    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85" pod "b6527b2f-0450-4b50-83ed-19262fd9671f" (UID: "b6527b2f-0450-4b50-83ed-19262fd9671f"). InnerVolumeSpecName "kube-api-access-p4l85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:31:51.710391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.710347    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6527b2f-0450-4b50-83ed-19262fd9671f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:51.710724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:51.710701    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4l85\" (UniqueName: \"kubernetes.io/projected/b6527b2f-0450-4b50-83ed-19262fd9671f-kube-api-access-p4l85\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:31:52.242678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.242644    5515 generic.go:356] "Generic (PLEG): container finished" podID="b6527b2f-0450-4b50-83ed-19262fd9671f" containerID="9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246" exitCode=0
Nov 03 03:31:52.243189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.242747    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm" event={"ID":"b6527b2f-0450-4b50-83ed-19262fd9671f","Type":"ContainerDied","Data":"9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246"}
Nov 03 03:31:52.243189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.242783    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm" event={"ID":"b6527b2f-0450-4b50-83ed-19262fd9671f","Type":"ContainerDied","Data":"c81f7f6a2ae99c60aaf4822d41aff514665dce5025f5027d2d3f99081d895023"}
Nov 03 03:31:52.243189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.242804    5515 scope.go:122] "RemoveContainer" containerID="9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246"
Nov 03 03:31:52.243189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.242879    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"
Nov 03 03:31:52.259122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.259103    5515 scope.go:122] "RemoveContainer" containerID="9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246"
Nov 03 03:31:52.259425 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:52.259408    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246\": container with ID starting with 9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246 not found: ID does not exist" containerID="9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246"
Nov 03 03:31:52.259508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.259430    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246"} err="failed to get container status \"9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246\": rpc error: code = NotFound desc = could not find container \"9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246\": container with ID starting with 9a233edae101b5d0d1fda81a65ab7fbfb278b8686a794a87801f05d9b14bf246 not found: ID does not exist"
Nov 03 03:31:52.441707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.441599    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"]
Nov 03 03:31:52.445774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.445734    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-398727-5b7c99f59-9xfdm"]
Nov 03 03:31:52.578641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.578184    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:52.755575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.755437    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:52.919576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.915913    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6527b2f-0450-4b50-83ed-19262fd9671f" path="/var/lib/kubelet/pods/b6527b2f-0450-4b50-83ed-19262fd9671f/volumes"
Nov 03 03:31:52.919576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:52.916722    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" path="/var/lib/kubelet/pods/b7102b72-a704-44f6-a6a4-dbb727d47ed1/volumes"
Nov 03 03:31:57.570321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.570288    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"]
Nov 03 03:31:57.571001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.570983    5515 cpu_manager.go:414] "containerMap: removing container" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy"
Nov 03 03:31:57.571049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571004    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy"
Nov 03 03:31:57.571049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571016    5515 cpu_manager.go:414] "containerMap: removing container" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy"
Nov 03 03:31:57.571049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571021    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy"
Nov 03 03:31:57.571049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571042    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy"
Nov 03 03:31:57.571049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571046    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571057    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571062    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571076    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b6527b2f-0450-4b50-83ed-19262fd9671f" containerName="payload-pre-processing"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571082    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6527b2f-0450-4b50-83ed-19262fd9671f" containerName="payload-pre-processing"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571089    5515 cpu_manager.go:414] "containerMap: removing container" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571094    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571186    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bb2648-08d7-436b-82fa-8a06538a04ae" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571197    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5995f7c2-777d-4518-ab7e-712a54ca57e4" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571202    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7102b72-a704-44f6-a6a4-dbb727d47ed1" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571208    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6527b2f-0450-4b50-83ed-19262fd9671f" containerName="payload-pre-processing"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571215    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="256bd96e-dc29-439c-9978-a0cc51c869c7" containerName="istio-proxy"
Nov 03 03:31:57.571228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.571224    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="095440b1-e9d8-4acc-90c5-d973820d82d2" containerName="istio-proxy"
Nov 03 03:31:57.576115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.576099    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.629385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.629351    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9902a0c2-serving-cert\""
Nov 03 03:31:57.633102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.633081    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.633194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.633172    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8np7l\" (UniqueName: \"kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.633319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.633297    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.633380 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.633342    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.685142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.685100    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"]
Nov 03 03:31:57.734947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.734913    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.735120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.734954    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.735120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.735065    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.735195 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:57.735134    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-serving-cert: secret "maas-api-e2e-mt-9902a0c2-serving-cert" not found
Nov 03 03:31:57.735230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.735144    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8np7l\" (UniqueName: \"kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.735230 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:31:57.735221    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls podName:d05c116a-060b-483a-a9af-9e2b9c7d2148 nodeName:}" failed. No retries permitted until 2026-11-03 03:31:58.235197244 +0000 UTC m=+1809.466089657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls") pod "maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" (UID: "d05c116a-060b-483a-a9af-9e2b9c7d2148") : secret "maas-api-e2e-mt-9902a0c2-serving-cert" not found
Nov 03 03:31:57.735694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.735674    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.737778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.737759    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.802785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.802748    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8np7l\" (UniqueName: \"kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:57.817305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.817270    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:57.969830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:57.969800    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:58.121574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:58.119413    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:31:58.245408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:58.245323    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:58.248241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:58.248213    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\") pod \"maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:58.490284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:58.490251    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:31:58.913935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:58.913903    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"]
Nov 03 03:31:58.914651 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:31:58.914627    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05c116a_060b_483a_a9af_9e2b9c7d2148.slice/crio-878fbba44c240fa99d40c856fce00c1f5766cb6d05d673069417991ab0aa8715 WatchSource:0}: Error finding container 878fbba44c240fa99d40c856fce00c1f5766cb6d05d673069417991ab0aa8715: Status 404 returned error can't find the container with id 878fbba44c240fa99d40c856fce00c1f5766cb6d05d673069417991ab0aa8715
Nov 03 03:31:58.916577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:58.916541    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:31:59.294516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:59.294437    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" event={"ID":"d05c116a-060b-483a-a9af-9e2b9c7d2148","Type":"ContainerStarted","Data":"878fbba44c240fa99d40c856fce00c1f5766cb6d05d673069417991ab0aa8715"}
Nov 03 03:31:59.339657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:31:59.339623    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:00.224342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.224309    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"]
Nov 03 03:32:00.228869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.228844    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.304336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.304300    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" event={"ID":"d05c116a-060b-483a-a9af-9e2b9c7d2148","Type":"ContainerStarted","Data":"39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87"}
Nov 03 03:32:00.304498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.304366    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:32:00.310022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.309994    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:00.334660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.334631    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"]
Nov 03 03:32:00.399695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.399661    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4a5156-openshift-default-dockercfg-q882m\""
Nov 03 03:32:00.445303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445259    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.445499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445357    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.445499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445419    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.445499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445447    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtvz\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.445723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445641    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.445723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445687    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.445903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.445882    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.447070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.447046    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.447297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.447276    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.543903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.543745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554168    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtvz\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554273    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554308    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554387    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554500    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554537    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554648    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554704    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.554744    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.556610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.555206    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.557287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.557064    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.557801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.557427    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.557801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.557536    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.559939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.558067    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.566387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.566037    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.567404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.567369    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.617465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.617430    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtvz\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.617844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.617821    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token\") pod \"e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") " pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:00.846223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:00.845840    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:01.338981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:01.338913    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" podStartSLOduration=3.165396432 podStartE2EDuration="4.338895556s" totalImagesPullingTime="1.173499124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:00.542989441 +0000 UTC m=+1811.773881854" watchObservedRunningTime="2026-11-03 03:32:01.338895556 +0000 UTC m=+1812.569787974"
Nov 03 03:32:01.340753 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:32:01.340729    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5127b2e0_c1fc_445c_a158_eea5dfab716e.slice/crio-6b87f91f41e5f55120af22cbbab6975d99d8c68e572162891a9093b5403fe373 WatchSource:0}: Error finding container 6b87f91f41e5f55120af22cbbab6975d99d8c68e572162891a9093b5403fe373: Status 404 returned error can't find the container with id 6b87f91f41e5f55120af22cbbab6975d99d8c68e572162891a9093b5403fe373
Nov 03 03:32:01.341698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:01.341672    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"]
Nov 03 03:32:01.343507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:01.343476    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:32:01.343590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:01.343571    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:32:01.343637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:01.343605    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:32:02.319527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:02.319485    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" event={"ID":"5127b2e0-c1fc-445c-a158-eea5dfab716e","Type":"ContainerStarted","Data":"5f9b8d89faea9a865bec652633b858cbfc1b3bfbe6af02a12dc8b8d60d5ca6ac"}
Nov 03 03:32:02.319750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:02.319605    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" event={"ID":"5127b2e0-c1fc-445c-a158-eea5dfab716e","Type":"ContainerStarted","Data":"6b87f91f41e5f55120af22cbbab6975d99d8c68e572162891a9093b5403fe373"}
Nov 03 03:32:02.545821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:02.545765    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podStartSLOduration=2.545749792 podStartE2EDuration="2.545749792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:02.492748342 +0000 UTC m=+1813.723640775" watchObservedRunningTime="2026-11-03 03:32:02.545749792 +0000 UTC m=+1813.776642210"
Nov 03 03:32:02.847404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:02.847363    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:02.853379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:02.853346    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:03.030841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:03.030789    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:03.326621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:03.326590    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:03.327601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:03.327584    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:32:05.308004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:05.307964    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:05.490342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:05.490304    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:05.638616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:05.638537    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:06.317659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:06.317183    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:32:08.252821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:08.252785    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:08.556778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:08.556667    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:08.770779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:08.770709    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:13.172439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.172395    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"]
Nov 03 03:32:13.177124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.177101    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.232792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.232759    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-687adc-serving-cert\""
Nov 03 03:32:13.285565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.285515    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.285757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.285660    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.285824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.285805    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.285900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.285880    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lsc\" (UniqueName: \"kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.286786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.286760    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"]
Nov 03 03:32:13.371438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.371382    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:13.387854 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.387824    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.388035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.387915    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lsc\" (UniqueName: \"kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.388129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.388056    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.388195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.388138    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.388359 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:13.388336    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-serving-cert: secret "maas-api-e2e-api-b-687adc-serving-cert" not found
Nov 03 03:32:13.388575 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:13.388542    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls podName:aadefec4-c215-4e51-93c0-489eee7b720c nodeName:}" failed. No retries permitted until 2026-11-03 03:32:13.88851804 +0000 UTC m=+1825.119410450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls") pod "maas-api-e2e-api-b-687adc-dcd886f74-wqbps" (UID: "aadefec4-c215-4e51-93c0-489eee7b720c") : secret "maas-api-e2e-api-b-687adc-serving-cert" not found
Nov 03 03:32:13.389078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.389053    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.390668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.390648    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.464632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.464160    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lsc\" (UniqueName: \"kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.566233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.566197    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:13.752592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.752450    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:13.895360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.895328    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:13.898061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:13.898019    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\") pod \"maas-api-e2e-api-b-687adc-dcd886f74-wqbps\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:14.091851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:14.091753    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:14.471325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:14.471297    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"]
Nov 03 03:32:14.472403 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:32:14.472376    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadefec4_c215_4e51_93c0_489eee7b720c.slice/crio-16f3e57ff5ab3f6dd3c29faae39b08f2c0f7df2c104532784b82c194327e8676 WatchSource:0}: Error finding container 16f3e57ff5ab3f6dd3c29faae39b08f2c0f7df2c104532784b82c194327e8676: Status 404 returned error can't find the container with id 16f3e57ff5ab3f6dd3c29faae39b08f2c0f7df2c104532784b82c194327e8676
Nov 03 03:32:15.452940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:15.452801    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps" event={"ID":"aadefec4-c215-4e51-93c0-489eee7b720c","Type":"ContainerStarted","Data":"5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393"}
Nov 03 03:32:15.452940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:15.452873    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps" event={"ID":"aadefec4-c215-4e51-93c0-489eee7b720c","Type":"ContainerStarted","Data":"16f3e57ff5ab3f6dd3c29faae39b08f2c0f7df2c104532784b82c194327e8676"}
Nov 03 03:32:15.453154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:15.452951    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:15.579368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:15.579264    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps" podStartSLOduration=1.7902087 podStartE2EDuration="2.579249081s" totalImagesPullingTime="789.040381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:15.578204631 +0000 UTC m=+1826.809097049" watchObservedRunningTime="2026-11-03 03:32:15.579249081 +0000 UTC m=+1826.810141499"
Nov 03 03:32:18.218574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:18.218480    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:18.369066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:18.369032    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:18.541025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:18.540949    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:20.985298 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:20.985254    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:32:21.471051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:21.471021    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:32:23.425867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:23.425831    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:27.656540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:27.656503    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:29.357833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:29.357802    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:29.505023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:29.504975    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:29.656537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:29.656396    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:31.148016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:31.147986    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:34.427058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:34.427020    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:38.268350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:38.268298    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:38.415937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:38.415900    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:39.278460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:39.278426    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:32:39.454153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:39.454122    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:40.518034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:40.517992    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:41.032268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:41.032235    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"]
Nov 03 03:32:41.032466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:41.032447    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" containerID="cri-o://ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd" gracePeriod=30
Nov 03 03:32:41.048951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:41.048915    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"]
Nov 03 03:32:41.049186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:41.049162    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy" containerID="cri-o://2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e" gracePeriod=30
Nov 03 03:32:41.137574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:41.119728    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:42.088029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:42.087987    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:42.766573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:42.756055    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:43.035532 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:43.035457    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:44.183576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:44.182367    5515 patch_prober.go:29] interesting pod/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:32:44.183576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:44.182432    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:32:44.846428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:44.846399    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:46.343975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.343946    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:32:46.378823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.378752    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:32:46.400963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.400927    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401016    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401117    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401229    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401315    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401345    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401371    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.401513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401400    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.401513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401412    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401459    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401488    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr\") pod \"036c8e7a-11e5-4fb6-9471-23698540e2ca\" (UID: \"036c8e7a-11e5-4fb6-9471-23698540e2ca\") "
Nov 03 03:32:46.401752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401648    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.401752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401715    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:32:46.401982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.401942    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.402390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.402365    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.402496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.402396    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.402496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.402413    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036c8e7a-11e5-4fb6-9471-23698540e2ca-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.402496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.402429    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.402496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.402445    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.403694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.403668    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:32:46.403910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.403885    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.404169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.404141    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:32:46.404285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.404267    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr" pod "036c8e7a-11e5-4fb6-9471-23698540e2ca" (UID: "036c8e7a-11e5-4fb6-9471-23698540e2ca"). InnerVolumeSpecName "kube-api-access-s5tlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:32:46.503430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503393    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.503640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503463    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.503640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503589    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.503640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503629    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.503791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503738    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.503791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503743    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.503791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503771    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.504017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503828    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.504017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503847    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.504017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503898    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.504017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.503943    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data\") pod \"01a74f5d-e683-4502-8550-241b634591db\" (UID: \"01a74f5d-e683-4502-8550-241b634591db\") "
Nov 03 03:32:46.504242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504091    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.504242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504149    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504486    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504704    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504722    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504734    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504746    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504760    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504772    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a74f5d-e683-4502-8550-241b634591db-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504784    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504795    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036c8e7a-11e5-4fb6-9471-23698540e2ca-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.505223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.504806    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5tlr\" (UniqueName: \"kubernetes.io/projected/036c8e7a-11e5-4fb6-9471-23698540e2ca-kube-api-access-s5tlr\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.507976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.507949    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:32:46.508091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.508007    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "kube-api-access-q6hr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:32:46.508091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.508050    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:32:46.508091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.508058    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token" pod "01a74f5d-e683-4502-8550-241b634591db" (UID: "01a74f5d-e683-4502-8550-241b634591db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:32:46.518903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.518878    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:46.606402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.606369    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.606583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.606406    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a74f5d-e683-4502-8550-241b634591db-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.606583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.606423    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hr5\" (UniqueName: \"kubernetes.io/projected/01a74f5d-e683-4502-8550-241b634591db-kube-api-access-q6hr5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.606583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.606441    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a74f5d-e683-4502-8550-241b634591db-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:32:46.683980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.683887    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:46.832259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:46.832222    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:47.066315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.066276    5515 generic.go:356] "Generic (PLEG): container finished" podID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerID="2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e" exitCode=0
Nov 03 03:32:47.066484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.066327    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"
Nov 03 03:32:47.066484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.066362    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" event={"ID":"036c8e7a-11e5-4fb6-9471-23698540e2ca","Type":"ContainerDied","Data":"2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e"}
Nov 03 03:32:47.066484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.066422    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn" event={"ID":"036c8e7a-11e5-4fb6-9471-23698540e2ca","Type":"ContainerDied","Data":"5e2dfe90e6182b1223f08714c7ec2840d96ac6f2c07a3c1954b7eef34664da1b"}
Nov 03 03:32:47.066484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.066450    5515 scope.go:122] "RemoveContainer" containerID="2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e"
Nov 03 03:32:47.068249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.068223    5515 generic.go:356] "Generic (PLEG): container finished" podID="01a74f5d-e683-4502-8550-241b634591db" containerID="ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd" exitCode=0
Nov 03 03:32:47.068367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.068279    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"
Nov 03 03:32:47.068367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.068295    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" event={"ID":"01a74f5d-e683-4502-8550-241b634591db","Type":"ContainerDied","Data":"ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd"}
Nov 03 03:32:47.068367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.068352    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx" event={"ID":"01a74f5d-e683-4502-8550-241b634591db","Type":"ContainerDied","Data":"0b2c21ab6080e5c2fc29f7d2611e18de5c77e8d97c8675759867f14fa2ed64b8"}
Nov 03 03:32:47.078142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.078120    5515 scope.go:122] "RemoveContainer" containerID="ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"
Nov 03 03:32:47.088130 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.088100    5515 scope.go:122] "RemoveContainer" containerID="2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e"
Nov 03 03:32:47.088453 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:47.088428    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e\": container with ID starting with 2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e not found: ID does not exist" containerID="2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e"
Nov 03 03:32:47.088536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.088457    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e"} err="failed to get container status \"2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e\": rpc error: code = NotFound desc = could not find container \"2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e\": container with ID starting with 2381aa98e7d3d52e6ff0837444650a6f850448a4c0aac1daa27e206170718e1e not found: ID does not exist"
Nov 03 03:32:47.088536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.088477    5515 scope.go:122] "RemoveContainer" containerID="ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"
Nov 03 03:32:47.088832 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:47.088806    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6\": container with ID starting with ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6 not found: ID does not exist" containerID="ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"
Nov 03 03:32:47.088898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.088845    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6"} err="failed to get container status \"ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6\": rpc error: code = NotFound desc = could not find container \"ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6\": container with ID starting with ccd7ea4418916176e34e931676f6608c7a84465408b4158180bfad6a6f8f50b6 not found: ID does not exist"
Nov 03 03:32:47.088898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.088865    5515 scope.go:122] "RemoveContainer" containerID="ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd"
Nov 03 03:32:47.098710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.098668    5515 scope.go:122] "RemoveContainer" containerID="ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd"
Nov 03 03:32:47.098950 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:47.098933    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd\": container with ID starting with ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd not found: ID does not exist" containerID="ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd"
Nov 03 03:32:47.099043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.098956    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd"} err="failed to get container status \"ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd\": rpc error: code = NotFound desc = could not find container \"ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd\": container with ID starting with ac79bc35bbad636378af838ecffe98bc2d0552035458aaef41879b546b4394dd not found: ID does not exist"
Nov 03 03:32:47.239047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.239000    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"]
Nov 03 03:32:47.289509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.289472    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-742fn"]
Nov 03 03:32:47.355482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.355398    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"]
Nov 03 03:32:47.360947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.360913    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-398727-openshift-default-56ff48bb55-rqsmx"]
Nov 03 03:32:47.403229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.403182    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:32:47.403386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.403242    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:32:47.545604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.545565    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:47.691053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:47.691015    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:48.897967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:48.897936    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a74f5d-e683-4502-8550-241b634591db" path="/var/lib/kubelet/pods/01a74f5d-e683-4502-8550-241b634591db/volumes"
Nov 03 03:32:48.898451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:48.898437    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" path="/var/lib/kubelet/pods/036c8e7a-11e5-4fb6-9471-23698540e2ca/volumes"
Nov 03 03:32:49.736337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:49.736250    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:49.977612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:49.977574    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:50.141575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.137166    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.159291    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"]
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.160368    5515 cpu_manager.go:414] "containerMap: removing container" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy"
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.160389    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy"
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.160444    5515 cpu_manager.go:414] "containerMap: removing container" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy"
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.160452    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy"
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.160628    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy"
Nov 03 03:32:50.163575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.160648    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a74f5d-e683-4502-8550-241b634591db" containerName="istio-proxy"
Nov 03 03:32:50.167572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.165615    5515 generic.go:356] "Generic (PLEG): container finished" podID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerID="3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37" exitCode=0
Nov 03 03:32:50.177597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.168419    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" event={"ID":"3bfe2231-d5e3-4ed6-88c9-07483aae2c43","Type":"ContainerDied","Data":"3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"}
Nov 03 03:32:50.177597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.168510    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.177597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.169905    5515 scope.go:122] "RemoveContainer" containerID="3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"
Nov 03 03:32:50.177597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.174380    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:32:50.177597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.174484    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:32:50.177597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.174537    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:32:50.183842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.183809    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.184034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.184019    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9jq\" (UniqueName: \"kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.216494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.216454    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"]
Nov 03 03:32:50.273574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.272950    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95ab58\""
Nov 03 03:32:50.273574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.273053    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95ab58-dockercfg-lkclj\""
Nov 03 03:32:50.306573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.303899    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9jq\" (UniqueName: \"kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.306573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.304021    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.306573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.305354    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.400576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.399660    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"]
Nov 03 03:32:50.400779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.400717    5515 cpu_manager.go:414] "containerMap: removing container" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy"
Nov 03 03:32:50.400779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.400737    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy"
Nov 03 03:32:50.417599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.401073    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="036c8e7a-11e5-4fb6-9471-23698540e2ca" containerName="istio-proxy"
Nov 03 03:32:50.417599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.416574    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.420834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.418615    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"]
Nov 03 03:32:50.420834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.419443    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvzg\" (UniqueName: \"kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg\") pod \"payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.420834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.419497    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.441441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.429260    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9jq\" (UniqueName: \"kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.495290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.494777    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:50.529639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.527948    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvzg\" (UniqueName: \"kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg\") pod \"payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.529639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.529512    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.531669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.530973    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.597657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.595686    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvzg\" (UniqueName: \"kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg\") pod \"payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.789517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.789422    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:50.897621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:50.897588    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"]
Nov 03 03:32:50.898290 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:32:50.898255    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feba1c5_8767_405f_9ce5_074146aeadf0.slice/crio-c35ada27cfc286072ef76314dc0e595f34f6234706e581c6a735ff3a247c107f WatchSource:0}: Error finding container c35ada27cfc286072ef76314dc0e595f34f6234706e581c6a735ff3a247c107f: Status 404 returned error can't find the container with id c35ada27cfc286072ef76314dc0e595f34f6234706e581c6a735ff3a247c107f
Nov 03 03:32:51.004759 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:32:51.004724    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:32:51.186572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.185923    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl" event={"ID":"6feba1c5-8767-405f-9ce5-074146aeadf0","Type":"ContainerStarted","Data":"eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729"}
Nov 03 03:32:51.186572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.185992    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl" event={"ID":"6feba1c5-8767-405f-9ce5-074146aeadf0","Type":"ContainerStarted","Data":"c35ada27cfc286072ef76314dc0e595f34f6234706e581c6a735ff3a247c107f"}
Nov 03 03:32:51.191967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.187634    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:32:51.192876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.192849    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" event={"ID":"3bfe2231-d5e3-4ed6-88c9-07483aae2c43","Type":"ContainerStarted","Data":"8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e"}
Nov 03 03:32:51.193922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.193899    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"]
Nov 03 03:32:51.195412 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:32:51.195388    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee494ea4_9f1c_4050_9449_f4918beb65d1.slice/crio-3fe0e85eb8eb624b11922d1add0fcbf9ae90504209e7357b7ef3b21a8a4e2de4 WatchSource:0}: Error finding container 3fe0e85eb8eb624b11922d1add0fcbf9ae90504209e7357b7ef3b21a8a4e2de4: Status 404 returned error can't find the container with id 3fe0e85eb8eb624b11922d1add0fcbf9ae90504209e7357b7ef3b21a8a4e2de4
Nov 03 03:32:51.306965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.306904    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl" podStartSLOduration=1.306891051 podStartE2EDuration="1.306891051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:51.305217641 +0000 UTC m=+1862.536110059" watchObservedRunningTime="2026-11-03 03:32:51.306891051 +0000 UTC m=+1862.537783468"
Nov 03 03:32:51.507919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.507820    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:32:51.508101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.508064    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 03:32:51.508182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:51.508114    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 03:32:52.201822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:52.201780    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6" event={"ID":"ee494ea4-9f1c-4050-9449-f4918beb65d1","Type":"ContainerStarted","Data":"d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe"}
Nov 03 03:32:52.202313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:52.201879    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6" event={"ID":"ee494ea4-9f1c-4050-9449-f4918beb65d1","Type":"ContainerStarted","Data":"3fe0e85eb8eb624b11922d1add0fcbf9ae90504209e7357b7ef3b21a8a4e2de4"}
Nov 03 03:32:52.202313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:52.202232    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:32:52.322764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:52.322707    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6" podStartSLOduration=2.322692335 podStartE2EDuration="2.322692335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:32:52.319795122 +0000 UTC m=+1863.550687542" watchObservedRunningTime="2026-11-03 03:32:52.322692335 +0000 UTC m=+1863.553584794"
Nov 03 03:32:52.513629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:52.511034    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 03:32:52.513629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:52.511102    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 03:32:53.507947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:53.507915    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 03:32:53.508364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:53.507971    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 03:32:53.800844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:53.800755    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:53.948468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:53.948431    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:54.507380 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:54.507349    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 03:32:54.507582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:54.507406    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 03:32:55.417371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:55.417330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:32:55.510085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:55.510021    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 03:32:55.510379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:55.510349    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 03:32:56.507965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:56.507920    5515 patch_prober.go:29] interesting pod/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 03:32:56.508396 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:56.507987    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 03:32:57.511732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:57.511702    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:32:57.512243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:57.512223    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:32:57.512792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:57.512775    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:32:58.331910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:32:58.331883    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:00.827869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:00.827825    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:03.211651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:03.211623    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:33:03.212058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:03.211687    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:33:05.915756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:05.915721    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"]
Nov 03 03:33:05.918871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:05.918854    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:05.980103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:05.980068    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4a5156-serving-cert\""
Nov 03 03:33:06.005083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.005048    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.005256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.005128    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvz5\" (UniqueName: \"kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.005256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.005160    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.005396 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.005367    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.033926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.033892    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"]
Nov 03 03:33:06.106965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.106923    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.107165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.107084    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.107242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.107185    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvz5\" (UniqueName: \"kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.107242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.107220    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.107430 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:06.107397    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-serving-cert: secret "maas-api-e2e-ipp-a-4a5156-serving-cert" not found
Nov 03 03:33:06.107519 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:06.107501    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls podName:195f97df-261f-4a58-8855-4b1fa447d7eb nodeName:}" failed. No retries permitted until 2026-11-03 03:33:06.607481765 +0000 UTC m=+1877.838374160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls") pod "maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" (UID: "195f97df-261f-4a58-8855-4b1fa447d7eb") : secret "maas-api-e2e-ipp-a-4a5156-serving-cert" not found
Nov 03 03:33:06.107803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.107779    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.108647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.108626    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:06.109909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.109879    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.195716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.195637    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvz5\" (UniqueName: \"kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.275933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.275886    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:06.430233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.430184    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:06.618156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.618116    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.620771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.620745    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:06.832684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:06.832646    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:07.210122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:07.210092    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"]
Nov 03 03:33:07.210798 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:33:07.210772    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f97df_261f_4a58_8855_4b1fa447d7eb.slice/crio-06a3fd479c347a936a3356ed915c889fd3c925b3bb4b05d41e530fa7e9f200e6 WatchSource:0}: Error finding container 06a3fd479c347a936a3356ed915c889fd3c925b3bb4b05d41e530fa7e9f200e6: Status 404 returned error can't find the container with id 06a3fd479c347a936a3356ed915c889fd3c925b3bb4b05d41e530fa7e9f200e6
Nov 03 03:33:07.315116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:07.315080    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" event={"ID":"195f97df-261f-4a58-8855-4b1fa447d7eb","Type":"ContainerStarted","Data":"06a3fd479c347a936a3356ed915c889fd3c925b3bb4b05d41e530fa7e9f200e6"}
Nov 03 03:33:07.773283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:07.773244    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:33:09.046158 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:09.046122    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:09.355474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:09.355384    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" event={"ID":"195f97df-261f-4a58-8855-4b1fa447d7eb","Type":"ContainerStarted","Data":"39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23"}
Nov 03 03:33:09.355704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:09.355531    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:10.388777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:10.388731    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:11.617574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:11.617481    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:15.369629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:15.369598    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:33:15.438157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:15.438122    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:15.491772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:15.491719    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" podStartSLOduration=9.453819377 podStartE2EDuration="10.491706403s" totalImagesPullingTime="1.037887026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:09.486100707 +0000 UTC m=+1880.716993126" watchObservedRunningTime="2026-11-03 03:33:15.491706403 +0000 UTC m=+1886.722598821"
Nov 03 03:33:16.179685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.179648    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"]
Nov 03 03:33:16.179905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.179856    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" containerID="cri-o://a2c096338f75705e58a3c5711bdd92b59d28fe1161fd0ad6b7448eb98b2531f1" gracePeriod=30
Nov 03 03:33:16.231927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.231889    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"]
Nov 03 03:33:16.232156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.232129    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy" containerID="cri-o://e9f34293ffa79dca918608174baa4346d08aad23f217ee021c7fdc10a4c0887a" gracePeriod=30
Nov 03 03:33:16.236223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.236190    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"]
Nov 03 03:33:16.236399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.236374    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" containerID="cri-o://e1c32524d1c0c940b9deca8cde4c9f132fe4303b0e089ef0579c3485ce9b318c" gracePeriod=30
Nov 03 03:33:16.281369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.281333    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"]
Nov 03 03:33:16.281634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.281605    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" containerID="cri-o://4ca1d5e6e47155ce4880c5147cfefdadd0b867f4483cf51578c593e4deba430a" gracePeriod=30
Nov 03 03:33:16.895322 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:16.895162    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:17.983728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:17.983689    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:33:17.984129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:17.983783    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:33:18.446568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:18.446524    5515 generic.go:356] "Generic (PLEG): container finished" podID="7cb11c27-bf88-4249-8603-f41209cc645b" containerID="f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938" exitCode=0
Nov 03 03:33:18.446808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:18.446602    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" event={"ID":"7cb11c27-bf88-4249-8603-f41209cc645b","Type":"ContainerDied","Data":"f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"}
Nov 03 03:33:18.447106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:18.447087    5515 scope.go:122] "RemoveContainer" containerID="f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"
Nov 03 03:33:18.448705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:18.448678    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:33:18.448774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:18.448747    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:33:18.448811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:18.448779    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:33:19.297791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:19.297757    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:33:19.298183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:19.297818    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:33:19.455370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:19.455338    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" event={"ID":"7cb11c27-bf88-4249-8603-f41209cc645b","Type":"ContainerStarted","Data":"020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a"}
Nov 03 03:33:19.634708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:19.634678    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:33:19.634906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:19.634836    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 03:33:19.634906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:19.634878    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 03:33:20.634443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:20.634414    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 03:33:20.634864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:20.634464    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 03:33:20.998361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:20.998277    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:33:20.998361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:20.998338    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:33:21.010152 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:21.010111    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:33:21.510773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.510460    5515 generic.go:356] "Generic (PLEG): container finished" podID="297031fd-1985-46c3-a747-64214ae62e96" containerID="a2c096338f75705e58a3c5711bdd92b59d28fe1161fd0ad6b7448eb98b2531f1" exitCode=0
Nov 03 03:33:21.510773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.510537    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" event={"ID":"297031fd-1985-46c3-a747-64214ae62e96","Type":"ContainerDied","Data":"a2c096338f75705e58a3c5711bdd92b59d28fe1161fd0ad6b7448eb98b2531f1"}
Nov 03 03:33:21.512596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.512569    5515 generic.go:356] "Generic (PLEG): container finished" podID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerID="4ca1d5e6e47155ce4880c5147cfefdadd0b867f4483cf51578c593e4deba430a" exitCode=0
Nov 03 03:33:21.512740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.512648    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" event={"ID":"046e4002-87ed-4a9b-a4a1-de4af35a9845","Type":"ContainerDied","Data":"4ca1d5e6e47155ce4880c5147cfefdadd0b867f4483cf51578c593e4deba430a"}
Nov 03 03:33:21.514436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.514417    5515 generic.go:356] "Generic (PLEG): container finished" podID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerID="e1c32524d1c0c940b9deca8cde4c9f132fe4303b0e089ef0579c3485ce9b318c" exitCode=0
Nov 03 03:33:21.514559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.514530    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" event={"ID":"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7","Type":"ContainerDied","Data":"e1c32524d1c0c940b9deca8cde4c9f132fe4303b0e089ef0579c3485ce9b318c"}
Nov 03 03:33:21.516825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.516800    5515 generic.go:356] "Generic (PLEG): container finished" podID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerID="e9f34293ffa79dca918608174baa4346d08aad23f217ee021c7fdc10a4c0887a" exitCode=0
Nov 03 03:33:21.516928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.516861    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" event={"ID":"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09","Type":"ContainerDied","Data":"e9f34293ffa79dca918608174baa4346d08aad23f217ee021c7fdc10a4c0887a"}
Nov 03 03:33:21.516928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.516897    5515 scope.go:122] "RemoveContainer" containerID="480a6f0238ae7c790427054b994fba11bc9ef2180b33aa4928ab4bc2461c78e5"
Nov 03 03:33:21.546583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.545823    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 03:33:21.546583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.545896    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 03:33:21.637643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.637607    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 03:33:21.638179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.638153    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 03:33:21.713582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.711823    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715016    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715059    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715108    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715158    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715252    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715365    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715430    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715493    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715586    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data\") pod \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\" (UID: \"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09\") "
Nov 03 03:33:21.716106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.715918    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.716877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.716352    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.716877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.716733    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.717088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.717049    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.717169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.717145    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:21.717494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.717470    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.721258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.721234    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:33:21.721699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.721674    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "kube-api-access-lxv9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:21.722672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.722648    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:21.722982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.722962    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy" pod "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" (UID: "5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.802669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.802096    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:33:21.807978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.807927    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.816811    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.816893    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.816979    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817007    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817054    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817094    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817198    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817230    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817262    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817293    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817380    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817410    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817487    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817523    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817593    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817622    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817664    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw\") pod \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\" (UID: \"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817741    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs\") pod \"046e4002-87ed-4a9b-a4a1-de4af35a9845\" (UID: \"046e4002-87ed-4a9b-a4a1-de4af35a9845\") "
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817088    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817277    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.817472    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818299    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818321    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818339    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818356    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818372    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.818371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818387    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818405    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxv9g\" (UniqueName: \"kubernetes.io/projected/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-kube-api-access-lxv9g\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818418    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818431    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818445    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818460    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818686    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.818870    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.819047    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.819461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.819346    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:21.819946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.819621    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.821504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.821477    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "kube-api-access-vnsvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:21.822345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.821869    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:21.822345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.822095    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:21.822345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.822300    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.822345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.822222    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:21.822941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.822897    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "kube-api-access-2l6gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:21.822941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.822901    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:33:21.823095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.822950    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:33:21.823367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.823334    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy" pod "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" (UID: "b7887d4e-c7a4-47c1-8b7c-41b52865d5c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.824866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.824833    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy" pod "046e4002-87ed-4a9b-a4a1-de4af35a9845" (UID: "046e4002-87ed-4a9b-a4a1-de4af35a9845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.919945    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.919979    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.919994    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920007    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920022    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920035    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920060    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920074    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046e4002-87ed-4a9b-a4a1-de4af35a9845-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920088    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920102    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l6gw\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-kube-api-access-2l6gw\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920117    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046e4002-87ed-4a9b-a4a1-de4af35a9845-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920131    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsvw\" (UniqueName: \"kubernetes.io/projected/046e4002-87ed-4a9b-a4a1-de4af35a9845-kube-api-access-vnsvw\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920144    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920156    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.920201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.920169    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046e4002-87ed-4a9b-a4a1-de4af35a9845-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:21.951345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:21.951323    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020518    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020689    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020727    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020779    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020827    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020884    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020920    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.020979    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021012    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d\") pod \"297031fd-1985-46c3-a747-64214ae62e96\" (UID: \"297031fd-1985-46c3-a747-64214ae62e96\") "
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021017    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021361    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021379    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021391    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021435    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021918    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021941    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021959    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021973    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.022346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.021989    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/297031fd-1985-46c3-a747-64214ae62e96-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.023782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.023619    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:22.023782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.023707    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "kube-api-access-7r77d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:22.024729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.024697    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:33:22.025123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.025094    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo" pod "297031fd-1985-46c3-a747-64214ae62e96" (UID: "297031fd-1985-46c3-a747-64214ae62e96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:33:22.089202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.089082    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:22.122875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.122846    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r77d\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-kube-api-access-7r77d\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.122875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.122871    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/297031fd-1985-46c3-a747-64214ae62e96-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.122875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.122880    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/297031fd-1985-46c3-a747-64214ae62e96-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.123092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.122888    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/297031fd-1985-46c3-a747-64214ae62e96-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:22.525214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.525141    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"
Nov 03 03:33:22.525214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.525190    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222" event={"ID":"5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09","Type":"ContainerDied","Data":"ebb1349758754bec2ee444c38b4e9cfc05cbcd700e30571834421d32d76bccfb"}
Nov 03 03:33:22.525445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.525248    5515 scope.go:122] "RemoveContainer" containerID="e9f34293ffa79dca918608174baa4346d08aad23f217ee021c7fdc10a4c0887a"
Nov 03 03:33:22.527006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.526980    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"
Nov 03 03:33:22.527125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.526980    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd" event={"ID":"297031fd-1985-46c3-a747-64214ae62e96","Type":"ContainerDied","Data":"1fe3ea4e81c5f257ef854faa5bab251a851c893fbe20ba57ead4b1fb2d149475"}
Nov 03 03:33:22.531655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.528510    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs" event={"ID":"046e4002-87ed-4a9b-a4a1-de4af35a9845","Type":"ContainerDied","Data":"2c7a5986f76c9fcfddaa3f18f1d02ae6ca14db7931657acfd8829d76d3d4b7bc"}
Nov 03 03:33:22.531655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.528566    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"
Nov 03 03:33:22.531655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.530161    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s" event={"ID":"b7887d4e-c7a4-47c1-8b7c-41b52865d5c7","Type":"ContainerDied","Data":"1046ed08ef04496f4cfd31eb33f56b2de5da6b815cf04031e1f0446321f88592"}
Nov 03 03:33:22.531655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.530186    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"
Nov 03 03:33:22.540301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.540271    5515 scope.go:122] "RemoveContainer" containerID="a2c096338f75705e58a3c5711bdd92b59d28fe1161fd0ad6b7448eb98b2531f1"
Nov 03 03:33:22.550531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.550513    5515 scope.go:122] "RemoveContainer" containerID="4ca1d5e6e47155ce4880c5147cfefdadd0b867f4483cf51578c593e4deba430a"
Nov 03 03:33:22.559493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.559475    5515 scope.go:122] "RemoveContainer" containerID="e1c32524d1c0c940b9deca8cde4c9f132fe4303b0e089ef0579c3485ce9b318c"
Nov 03 03:33:22.634633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.634599    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 03:33:22.634768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.634653    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 03:33:22.703543 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.703508    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"]
Nov 03 03:33:22.752892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.752866    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-qbgpd"]
Nov 03 03:33:22.817720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.817621    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"]
Nov 03 03:33:22.821518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.821492    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-24cvs"]
Nov 03 03:33:22.896223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.896188    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" path="/var/lib/kubelet/pods/046e4002-87ed-4a9b-a4a1-de4af35a9845/volumes"
Nov 03 03:33:22.896799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.896783    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="297031fd-1985-46c3-a747-64214ae62e96" path="/var/lib/kubelet/pods/297031fd-1985-46c3-a747-64214ae62e96/volumes"
Nov 03 03:33:22.949491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.949461    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"]
Nov 03 03:33:22.952785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:22.952763    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-dl222"]
Nov 03 03:33:23.089647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:23.089098    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"]
Nov 03 03:33:23.093562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:23.093524    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-9fw5s"]
Nov 03 03:33:23.635087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:23.635055    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 03:33:23.635277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:23.635109    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 03:33:24.635024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:24.634992    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 03:33:24.635399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:24.635042    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 03:33:24.895274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:24.895192    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" path="/var/lib/kubelet/pods/5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09/volumes"
Nov 03 03:33:24.895692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:24.895678    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" path="/var/lib/kubelet/pods/b7887d4e-c7a4-47c1-8b7c-41b52865d5c7/volumes"
Nov 03 03:33:25.638927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:25.638898    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:33:25.639300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:25.639232    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:33:25.639945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:25.639925    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:33:26.265389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:26.265355    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:27.408261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:27.408224    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:27.594966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:27.594936    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:32.522253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:32.522216    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:34.028343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:34.028309    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:34.367719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:34.367683    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:33:34.367906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:34.367748    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:33:35.156808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:35.156772    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"]
Nov 03 03:33:35.516010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:35.515941    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:35.660287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:35.660259    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:36.681237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:36.681107    5515 generic.go:356] "Generic (PLEG): container finished" podID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerID="5f9b8d89faea9a865bec652633b858cbfc1b3bfbe6af02a12dc8b8d60d5ca6ac" exitCode=0
Nov 03 03:33:36.682162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:36.681922    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" event={"ID":"5127b2e0-c1fc-445c-a158-eea5dfab716e","Type":"ContainerDied","Data":"5f9b8d89faea9a865bec652633b858cbfc1b3bfbe6af02a12dc8b8d60d5ca6ac"}
Nov 03 03:33:36.683102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:36.683086    5515 scope.go:122] "RemoveContainer" containerID="5f9b8d89faea9a865bec652633b858cbfc1b3bfbe6af02a12dc8b8d60d5ca6ac"
Nov 03 03:33:36.689621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:36.687561    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:33:36.689621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:36.687660    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:33:36.689621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:36.687703    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:33:37.697312 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:37.697277    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" event={"ID":"5127b2e0-c1fc-445c-a158-eea5dfab716e","Type":"ContainerStarted","Data":"6db15ebd10f13a30e8377dba12c8994cc490f6d4e30dbd581cdc446a3099ea10"}
Nov 03 03:33:37.741922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:37.741889    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:37.847025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:37.846997    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:33:37.847196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:37.847094    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 03 03:33:37.847196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:37.847129    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 03 03:33:38.846791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:38.846756    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 03 03:33:38.847200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:38.846812    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 03 03:33:39.847257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:39.847223    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 03 03:33:39.847687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:39.847283    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 03 03:33:39.972109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:39.972073    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:40.157919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.157882    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt" podUID="f0a5132a-039c-480e-a4c0-9a36427eb20c" containerName="payload-processing" containerID="cri-o://727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399" gracePeriod=25
Nov 03 03:33:40.495664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.495638    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:33:40.608696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.608666    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8\" (UniqueName: \"kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8\") pod \"f0a5132a-039c-480e-a4c0-9a36427eb20c\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") "
Nov 03 03:33:40.608868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.608707    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume\") pod \"f0a5132a-039c-480e-a4c0-9a36427eb20c\" (UID: \"f0a5132a-039c-480e-a4c0-9a36427eb20c\") "
Nov 03 03:33:40.609095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.609074    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume" pod "f0a5132a-039c-480e-a4c0-9a36427eb20c" (UID: "f0a5132a-039c-480e-a4c0-9a36427eb20c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:40.610980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.610958    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8" pod "f0a5132a-039c-480e-a4c0-9a36427eb20c" (UID: "f0a5132a-039c-480e-a4c0-9a36427eb20c"). InnerVolumeSpecName "kube-api-access-pftd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:40.710510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.710425    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftd8\" (UniqueName: \"kubernetes.io/projected/f0a5132a-039c-480e-a4c0-9a36427eb20c-kube-api-access-pftd8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:40.710510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.710456    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0a5132a-039c-480e-a4c0-9a36427eb20c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:40.717750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.717721    5515 generic.go:356] "Generic (PLEG): container finished" podID="f0a5132a-039c-480e-a4c0-9a36427eb20c" containerID="727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399" exitCode=0
Nov 03 03:33:40.717879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.717767    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"
Nov 03 03:33:40.717879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.717780    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt" event={"ID":"f0a5132a-039c-480e-a4c0-9a36427eb20c","Type":"ContainerDied","Data":"727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399"}
Nov 03 03:33:40.717879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.717827    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt" event={"ID":"f0a5132a-039c-480e-a4c0-9a36427eb20c","Type":"ContainerDied","Data":"5b12f62278c8974996c096551ed8a1840bdc4e44ae098acd86d06fc74ca07611"}
Nov 03 03:33:40.717879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.717847    5515 scope.go:122] "RemoveContainer" containerID="727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399"
Nov 03 03:33:40.727841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.727826    5515 scope.go:122] "RemoveContainer" containerID="727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399"
Nov 03 03:33:40.728121 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:40.728102    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399\": container with ID starting with 727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399 not found: ID does not exist" containerID="727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399"
Nov 03 03:33:40.728173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.728127    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399"} err="failed to get container status \"727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399\": rpc error: code = NotFound desc = could not find container \"727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399\": container with ID starting with 727d8ea06eb16821188f0a8c91c88741ede53d4b466be0c01c738d35dde51399 not found: ID does not exist"
Nov 03 03:33:40.846926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.846891    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 03 03:33:40.847106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.846941    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 03 03:33:40.901689 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.901652    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"]
Nov 03 03:33:40.906095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:40.906071    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-64c59d-5d7d848757-qq4gt"]
Nov 03 03:33:41.847129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:41.847098    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 03 03:33:41.847314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:41.847149    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 03 03:33:41.883572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:41.883514    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:42.847251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:42.847208    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 03 03:33:42.847646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:42.847272    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 03 03:33:42.894782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:42.894751    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a5132a-039c-480e-a4c0-9a36427eb20c" path="/var/lib/kubelet/pods/f0a5132a-039c-480e-a4c0-9a36427eb20c/volumes"
Nov 03 03:33:43.116626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:43.116595    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:43.850939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:43.850912    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:33:43.851461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:43.851440    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:33:43.852164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:43.852146    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:33:44.387307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:44.387275    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:45.403357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:45.403325    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:33:47.695805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.695764    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"]
Nov 03 03:33:47.696645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696623    5515 cpu_manager.go:414] "containerMap: removing container" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy"
Nov 03 03:33:47.696645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696645    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696664    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f0a5132a-039c-480e-a4c0-9a36427eb20c" containerName="payload-processing"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696671    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a5132a-039c-480e-a4c0-9a36427eb20c" containerName="payload-processing"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696687    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696694    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696706    5515 cpu_manager.go:414] "containerMap: removing container" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696714    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696737    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696743    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696776    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy"
Nov 03 03:33:47.696826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696784    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy"
Nov 03 03:33:47.697315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696942    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="046e4002-87ed-4a9b-a4a1-de4af35a9845" containerName="istio-proxy"
Nov 03 03:33:47.697315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696960    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a5132a-039c-480e-a4c0-9a36427eb20c" containerName="payload-processing"
Nov 03 03:33:47.697315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696972    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy"
Nov 03 03:33:47.697315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696982    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="297031fd-1985-46c3-a747-64214ae62e96" containerName="istio-proxy"
Nov 03 03:33:47.697315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.696993    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb50507-ebc4-4f1f-a847-8d3f2ebc0a09" containerName="istio-proxy"
Nov 03 03:33:47.697315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.697006    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7887d4e-c7a4-47c1-8b7c-41b52865d5c7" containerName="istio-proxy"
Nov 03 03:33:47.702462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.702441    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.760055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.760011    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"]
Nov 03 03:33:47.794625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.794582    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtsgs\" (UniqueName: \"kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.794812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.794696    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.794812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.794753    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.794917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.794901    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.806779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.806749    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-95ab58-serving-cert\""
Nov 03 03:33:47.892079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.892049    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:47.895713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.895688    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.895849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.895776    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsgs\" (UniqueName: \"kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.895849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.895830    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.895977 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:47.895844    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-serving-cert: secret "maas-api-e2e-ipp-a-95ab58-serving-cert" not found
Nov 03 03:33:47.895977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.895882    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.895977 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:47.895907    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls podName:0f0a70da-3b42-4618-af6e-ead7001835af nodeName:}" failed. No retries permitted until 2026-11-03 03:33:48.395888364 +0000 UTC m=+1919.626780779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls") pod "maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" (UID: "0f0a70da-3b42-4618-af6e-ead7001835af") : secret "maas-api-e2e-ipp-a-95ab58-serving-cert" not found
Nov 03 03:33:47.896511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.896494    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.898539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.898517    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:47.989229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:47.989147    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsgs\" (UniqueName: \"kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:48.056901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:48.056840    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:48.203830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:48.203795    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:48.410061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:48.410023    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:48.412624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:48.412599    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:48.616137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:48.616107    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:48.995080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:48.995051    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"]
Nov 03 03:33:48.996449 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:33:48.996415    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0a70da_3b42_4618_af6e_ead7001835af.slice/crio-e93052e3547c9c1b139ca66dcf15cecc1e887499f5ec6fdcb922ea278b7ae821 WatchSource:0}: Error finding container e93052e3547c9c1b139ca66dcf15cecc1e887499f5ec6fdcb922ea278b7ae821: Status 404 returned error can't find the container with id e93052e3547c9c1b139ca66dcf15cecc1e887499f5ec6fdcb922ea278b7ae821
Nov 03 03:33:49.779475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:49.779441    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" event={"ID":"0f0a70da-3b42-4618-af6e-ead7001835af","Type":"ContainerStarted","Data":"e93052e3547c9c1b139ca66dcf15cecc1e887499f5ec6fdcb922ea278b7ae821"}
Nov 03 03:33:50.812610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:50.807195    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" event={"ID":"0f0a70da-3b42-4618-af6e-ead7001835af","Type":"ContainerStarted","Data":"84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09"}
Nov 03 03:33:50.812610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:50.810224    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:50.879615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:50.879586    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:50.931077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:50.931009    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" podStartSLOduration=3.146332091 podStartE2EDuration="3.930993273s" totalImagesPullingTime="784.661182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:33:50.929595611 +0000 UTC m=+1922.160488029" watchObservedRunningTime="2026-11-03 03:33:50.930993273 +0000 UTC m=+1922.161885720"
Nov 03 03:33:50.986822 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:50.986782    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:33:51.314389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:51.314348    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"]
Nov 03 03:33:51.427001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:51.426966    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"]
Nov 03 03:33:51.676216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:51.676186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:51.824723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:51.824687    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:53.034792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:53.034761    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:53.182750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:53.182715    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:53.330011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:53.329939    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:53.734440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:53.734409    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"]
Nov 03 03:33:53.845942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:53.845896    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"]
Nov 03 03:33:54.095705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:54.095625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:54.302141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:54.302106    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:56.314692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.314627    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t" podUID="4d0f56d0-f2da-4506-bda0-9b971be15163" containerName="payload-processing" containerID="cri-o://0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb" gracePeriod=25
Nov 03 03:33:56.427302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.427263    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h" podUID="cbfaf6c4-1871-464a-8839-76c4100c7e79" containerName="payload-pre-processing" containerID="cri-o://dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f" gracePeriod=25
Nov 03 03:33:56.596278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.596235    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:33:56.711433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.711409    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:33:56.780451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.780419    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982\" (UniqueName: \"kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982\") pod \"4d0f56d0-f2da-4506-bda0-9b971be15163\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") "
Nov 03 03:33:56.780663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.780589    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume\") pod \"4d0f56d0-f2da-4506-bda0-9b971be15163\" (UID: \"4d0f56d0-f2da-4506-bda0-9b971be15163\") "
Nov 03 03:33:56.780933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.780902    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume" pod "4d0f56d0-f2da-4506-bda0-9b971be15163" (UID: "4d0f56d0-f2da-4506-bda0-9b971be15163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:56.782849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.782831    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982" pod "4d0f56d0-f2da-4506-bda0-9b971be15163" (UID: "4d0f56d0-f2da-4506-bda0-9b971be15163"). InnerVolumeSpecName "kube-api-access-g4982". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:56.793097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.793068    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:56.878378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.878350    5515 generic.go:356] "Generic (PLEG): container finished" podID="4d0f56d0-f2da-4506-bda0-9b971be15163" containerID="0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb" exitCode=0
Nov 03 03:33:56.878599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.878397    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"
Nov 03 03:33:56.878599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.878432    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t" event={"ID":"4d0f56d0-f2da-4506-bda0-9b971be15163","Type":"ContainerDied","Data":"0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb"}
Nov 03 03:33:56.878599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.878495    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t" event={"ID":"4d0f56d0-f2da-4506-bda0-9b971be15163","Type":"ContainerDied","Data":"9d120c0ed45c15877e52a616d4a95570f9d930d14a12cb7270a13af03a485f8d"}
Nov 03 03:33:56.878599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.878518    5515 scope.go:122] "RemoveContainer" containerID="0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb"
Nov 03 03:33:56.879858 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.879840    5515 generic.go:356] "Generic (PLEG): container finished" podID="cbfaf6c4-1871-464a-8839-76c4100c7e79" containerID="dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f" exitCode=0
Nov 03 03:33:56.879941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.879889    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"
Nov 03 03:33:56.880009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.879927    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h" event={"ID":"cbfaf6c4-1871-464a-8839-76c4100c7e79","Type":"ContainerDied","Data":"dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f"}
Nov 03 03:33:56.880009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.879991    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h" event={"ID":"cbfaf6c4-1871-464a-8839-76c4100c7e79","Type":"ContainerDied","Data":"1cca9820282c05583abaaf53161609628eb664d7e7770d81c39d9d7fef1adf2b"}
Nov 03 03:33:56.881571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.881539    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume\") pod \"cbfaf6c4-1871-464a-8839-76c4100c7e79\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") "
Nov 03 03:33:56.881706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.881692    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx\" (UniqueName: \"kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx\") pod \"cbfaf6c4-1871-464a-8839-76c4100c7e79\" (UID: \"cbfaf6c4-1871-464a-8839-76c4100c7e79\") "
Nov 03 03:33:56.881972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.881947    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume" pod "cbfaf6c4-1871-464a-8839-76c4100c7e79" (UID: "cbfaf6c4-1871-464a-8839-76c4100c7e79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:56.882156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.882138    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbfaf6c4-1871-464a-8839-76c4100c7e79-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:56.882212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.882162    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4982\" (UniqueName: \"kubernetes.io/projected/4d0f56d0-f2da-4506-bda0-9b971be15163-kube-api-access-g4982\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:56.882212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.882178    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0f56d0-f2da-4506-bda0-9b971be15163-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:56.884203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.884178    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx" pod "cbfaf6c4-1871-464a-8839-76c4100c7e79" (UID: "cbfaf6c4-1871-464a-8839-76c4100c7e79"). InnerVolumeSpecName "kube-api-access-nwltx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:56.889220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.889202    5515 scope.go:122] "RemoveContainer" containerID="0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb"
Nov 03 03:33:56.889507 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:56.889490    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb\": container with ID starting with 0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb not found: ID does not exist" containerID="0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb"
Nov 03 03:33:56.889587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.889512    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb"} err="failed to get container status \"0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb\": rpc error: code = NotFound desc = could not find container \"0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb\": container with ID starting with 0698e953319f63bba7aa58685c8f24afa100ab59a24b170e39a9912bd0dbdceb not found: ID does not exist"
Nov 03 03:33:56.889587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.889528    5515 scope.go:122] "RemoveContainer" containerID="dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f"
Nov 03 03:33:56.899098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.899082    5515 scope.go:122] "RemoveContainer" containerID="dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f"
Nov 03 03:33:56.899354 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:56.899335    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f\": container with ID starting with dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f not found: ID does not exist" containerID="dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f"
Nov 03 03:33:56.899401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.899363    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f"} err="failed to get container status \"dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f\": rpc error: code = NotFound desc = could not find container \"dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f\": container with ID starting with dbc181ffacc86d8ad3ada8e9b3b820d40142a39e1bf7e37883c9eb90c7586e8f not found: ID does not exist"
Nov 03 03:33:56.983265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:56.983238    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwltx\" (UniqueName: \"kubernetes.io/projected/cbfaf6c4-1871-464a-8839-76c4100c7e79-kube-api-access-nwltx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:57.100280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:57.100248    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"]
Nov 03 03:33:57.102840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:57.102813    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-372186-85995b8f6f-hxr9t"]
Nov 03 03:33:57.297069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:57.296983    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"]
Nov 03 03:33:57.301607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:57.301581    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-372186-7fb54dcdc9-qfj5h"]
Nov 03 03:33:57.805120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:57.805082    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:57.845727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:57.845700    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:33:58.634601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.634533    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:58.739577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.738619    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l" podUID="87980c56-7904-4dcb-8297-36cfd18adc3a" containerName="payload-processing" containerID="cri-o://7d119f6fa63746454a10171fe2031ddef843a2413a289ebdc2b4ef379e25dc58" gracePeriod=25
Nov 03 03:33:58.846636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.846590    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz" podUID="0ce923f5-be32-4b99-bcf7-1f0084f8c81a" containerName="payload-pre-processing" containerID="cri-o://4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419" gracePeriod=25
Nov 03 03:33:58.903362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.903267    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0f56d0-f2da-4506-bda0-9b971be15163" path="/var/lib/kubelet/pods/4d0f56d0-f2da-4506-bda0-9b971be15163/volumes"
Nov 03 03:33:58.903895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.903869    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfaf6c4-1871-464a-8839-76c4100c7e79" path="/var/lib/kubelet/pods/cbfaf6c4-1871-464a-8839-76c4100c7e79/volumes"
Nov 03 03:33:58.904052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.903977    5515 generic.go:356] "Generic (PLEG): container finished" podID="87980c56-7904-4dcb-8297-36cfd18adc3a" containerID="7d119f6fa63746454a10171fe2031ddef843a2413a289ebdc2b4ef379e25dc58" exitCode=0
Nov 03 03:33:58.904294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.904271    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l" event={"ID":"87980c56-7904-4dcb-8297-36cfd18adc3a","Type":"ContainerDied","Data":"7d119f6fa63746454a10171fe2031ddef843a2413a289ebdc2b4ef379e25dc58"}
Nov 03 03:33:58.942289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.942262    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"]
Nov 03 03:33:58.942497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:58.942477    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" podUID="d05c116a-060b-483a-a9af-9e2b9c7d2148" containerName="maas-api" containerID="cri-o://39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87" gracePeriod=30
Nov 03 03:33:59.047646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.047476    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:59.137860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.137831    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:33:59.312178 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.312146    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:33:59.335297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.335262    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm\" (UniqueName: \"kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm\") pod \"87980c56-7904-4dcb-8297-36cfd18adc3a\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") "
Nov 03 03:33:59.335506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.335443    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume\") pod \"87980c56-7904-4dcb-8297-36cfd18adc3a\" (UID: \"87980c56-7904-4dcb-8297-36cfd18adc3a\") "
Nov 03 03:33:59.336291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.336262    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume" pod "87980c56-7904-4dcb-8297-36cfd18adc3a" (UID: "87980c56-7904-4dcb-8297-36cfd18adc3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:59.338829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.338802    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm" pod "87980c56-7904-4dcb-8297-36cfd18adc3a" (UID: "87980c56-7904-4dcb-8297-36cfd18adc3a"). InnerVolumeSpecName "kube-api-access-lx8jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:59.414813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.414723    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:33:59.448621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.444054    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2\" (UniqueName: \"kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2\") pod \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") "
Nov 03 03:33:59.448621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.444245    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume\") pod \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\" (UID: \"0ce923f5-be32-4b99-bcf7-1f0084f8c81a\") "
Nov 03 03:33:59.448621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.444909    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87980c56-7904-4dcb-8297-36cfd18adc3a-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.448621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.444934    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8jm\" (UniqueName: \"kubernetes.io/projected/87980c56-7904-4dcb-8297-36cfd18adc3a-kube-api-access-lx8jm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.461660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.456147    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume" pod "0ce923f5-be32-4b99-bcf7-1f0084f8c81a" (UID: "0ce923f5-be32-4b99-bcf7-1f0084f8c81a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:59.463294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.462739    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2" pod "0ce923f5-be32-4b99-bcf7-1f0084f8c81a" (UID: "0ce923f5-be32-4b99-bcf7-1f0084f8c81a"). InnerVolumeSpecName "kube-api-access-pbtd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:59.545903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.545859    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca\") pod \"d05c116a-060b-483a-a9af-9e2b9c7d2148\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") "
Nov 03 03:33:59.546085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.545942    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\") pod \"d05c116a-060b-483a-a9af-9e2b9c7d2148\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") "
Nov 03 03:33:59.546214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.546195    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs\") pod \"d05c116a-060b-483a-a9af-9e2b9c7d2148\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") "
Nov 03 03:33:59.546284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.546218    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca" pod "d05c116a-060b-483a-a9af-9e2b9c7d2148" (UID: "d05c116a-060b-483a-a9af-9e2b9c7d2148"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:33:59.546284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.546272    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l\" (UniqueName: \"kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l\") pod \"d05c116a-060b-483a-a9af-9e2b9c7d2148\" (UID: \"d05c116a-060b-483a-a9af-9e2b9c7d2148\") "
Nov 03 03:33:59.547151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.547120    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.547151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.547156    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05c116a-060b-483a-a9af-9e2b9c7d2148-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.547358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.547172    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtd2\" (UniqueName: \"kubernetes.io/projected/0ce923f5-be32-4b99-bcf7-1f0084f8c81a-kube-api-access-pbtd2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.548369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.548348    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls" pod "d05c116a-060b-483a-a9af-9e2b9c7d2148" (UID: "d05c116a-060b-483a-a9af-9e2b9c7d2148"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:33:59.548449 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.548372    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs" pod "d05c116a-060b-483a-a9af-9e2b9c7d2148" (UID: "d05c116a-060b-483a-a9af-9e2b9c7d2148"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:33:59.548609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.548591    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l" pod "d05c116a-060b-483a-a9af-9e2b9c7d2148" (UID: "d05c116a-060b-483a-a9af-9e2b9c7d2148"). InnerVolumeSpecName "kube-api-access-8np7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:33:59.648616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.648583    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.648616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.648609    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05c116a-060b-483a-a9af-9e2b9c7d2148-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.648616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.648622    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8np7l\" (UniqueName: \"kubernetes.io/projected/d05c116a-060b-483a-a9af-9e2b9c7d2148-kube-api-access-8np7l\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:33:59.956576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.940671    5515 generic.go:356] "Generic (PLEG): container finished" podID="0ce923f5-be32-4b99-bcf7-1f0084f8c81a" containerID="4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419" exitCode=0
Nov 03 03:33:59.956576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.940814    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz" event={"ID":"0ce923f5-be32-4b99-bcf7-1f0084f8c81a","Type":"ContainerDied","Data":"4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419"}
Nov 03 03:33:59.956576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.940866    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz" event={"ID":"0ce923f5-be32-4b99-bcf7-1f0084f8c81a","Type":"ContainerDied","Data":"a261edd17da4332acf1e6148ee82d2d7587bf75b02811dca70cb5f611712f098"}
Nov 03 03:33:59.956576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.940895    5515 scope.go:122] "RemoveContainer" containerID="4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419"
Nov 03 03:33:59.956576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.941047    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"
Nov 03 03:33:59.956576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.948711    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:33:59.970574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.969511    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"
Nov 03 03:33:59.982787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.979674    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l" event={"ID":"87980c56-7904-4dcb-8297-36cfd18adc3a","Type":"ContainerDied","Data":"1ea49ee046b6076765148f2b615c8d11492560270949be4baf24a81eb125b6c1"}
Nov 03 03:33:59.982787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.980954    5515 generic.go:356] "Generic (PLEG): container finished" podID="d05c116a-060b-483a-a9af-9e2b9c7d2148" containerID="39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87" exitCode=0
Nov 03 03:33:59.982787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.981062    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" event={"ID":"d05c116a-060b-483a-a9af-9e2b9c7d2148","Type":"ContainerDied","Data":"39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87"}
Nov 03 03:33:59.982787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.981109    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn" event={"ID":"d05c116a-060b-483a-a9af-9e2b9c7d2148","Type":"ContainerDied","Data":"878fbba44c240fa99d40c856fce00c1f5766cb6d05d673069417991ab0aa8715"}
Nov 03 03:33:59.982787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.981197    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"
Nov 03 03:33:59.996037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.996001    5515 scope.go:122] "RemoveContainer" containerID="4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419"
Nov 03 03:33:59.996786 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:33:59.996588    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419\": container with ID starting with 4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419 not found: ID does not exist" containerID="4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419"
Nov 03 03:33:59.996786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.996633    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419"} err="failed to get container status \"4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419\": rpc error: code = NotFound desc = could not find container \"4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419\": container with ID starting with 4b7be9104a07858cd28eda502972f80d5a679cc0384a2b682e7c780e5edb5419 not found: ID does not exist"
Nov 03 03:33:59.996786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:33:59.996659    5515 scope.go:122] "RemoveContainer" containerID="7d119f6fa63746454a10171fe2031ddef843a2413a289ebdc2b4ef379e25dc58"
Nov 03 03:34:00.008946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.008928    5515 scope.go:122] "RemoveContainer" containerID="39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87"
Nov 03 03:34:00.020600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.020509    5515 scope.go:122] "RemoveContainer" containerID="39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87"
Nov 03 03:34:00.020893 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:00.020859    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87\": container with ID starting with 39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87 not found: ID does not exist" containerID="39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87"
Nov 03 03:34:00.021048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.020903    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87"} err="failed to get container status \"39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87\": rpc error: code = NotFound desc = could not find container \"39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87\": container with ID starting with 39618074dcbd8b9fa03f5ee4c986c5ac07376e6eec57908b0e87f9cd4572aa87 not found: ID does not exist"
Nov 03 03:34:00.116054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.116021    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"]
Nov 03 03:34:00.119984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.119956    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9902a0c2-649d68fc4c-8ms6l"]
Nov 03 03:34:00.231830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.231799    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"]
Nov 03 03:34:00.238021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.237999    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9902a0c2-7b9bbb7767-q8rsn"]
Nov 03 03:34:00.348798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.348764    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"]
Nov 03 03:34:00.353559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.353515    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9902a0c2-55d9b849d7-hgqvz"]
Nov 03 03:34:00.827894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.827864    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:00.916053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.916008    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce923f5-be32-4b99-bcf7-1f0084f8c81a" path="/var/lib/kubelet/pods/0ce923f5-be32-4b99-bcf7-1f0084f8c81a/volumes"
Nov 03 03:34:00.916670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.916649    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87980c56-7904-4dcb-8297-36cfd18adc3a" path="/var/lib/kubelet/pods/87980c56-7904-4dcb-8297-36cfd18adc3a/volumes"
Nov 03 03:34:00.917097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:00.917082    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05c116a-060b-483a-a9af-9e2b9c7d2148" path="/var/lib/kubelet/pods/d05c116a-060b-483a-a9af-9e2b9c7d2148/volumes"
Nov 03 03:34:04.770360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:04.770324    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:04.917116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:04.917076    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:05.711735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:05.711702    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:06.072830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:06.072744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:06.219935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:06.219902    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:06.372366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:06.372336    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:06.864457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:06.864420    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:06.866716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:06.866695    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"]
Nov 03 03:34:06.866932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:06.866913    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy" containerID="cri-o://9963df0cece920f5d612b8ae406e3e3d24c558845049c0a67f2168160ed048c0" gracePeriod=30
Nov 03 03:34:07.743011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:07.742980    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:07.896650 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:07.895826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:08.337752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:08.337718    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"]
Nov 03 03:34:08.337945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:08.337928    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy" containerID="cri-o://8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d" gracePeriod=30
Nov 03 03:34:08.345994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:08.345968    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:08.856190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:08.856159    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:09.025203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:09.025169    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:11.092210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:11.092179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:12.074636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.074598    5515 generic.go:356] "Generic (PLEG): container finished" podID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerID="9963df0cece920f5d612b8ae406e3e3d24c558845049c0a67f2168160ed048c0" exitCode=0
Nov 03 03:34:12.074836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.074675    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" event={"ID":"a4bbf043-42ca-4359-92b7-fb8ec834da65","Type":"ContainerDied","Data":"9963df0cece920f5d612b8ae406e3e3d24c558845049c0a67f2168160ed048c0"}
Nov 03 03:34:12.074836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.074792    5515 scope.go:122] "RemoveContainer" containerID="28a63ff1b29021d3865b7fb5a7a23ef21a9653539922bc6b36ab44b80f70419d"
Nov 03 03:34:12.131930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.131880    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:34:12.325867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.325829    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.326153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.326132    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.326360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.326346    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.326571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.326536    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.326731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.326714    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.326867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.326852    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.327654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.327270    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:12.327813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.327507    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:12.327928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.327521    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:12.328100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.328084    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.328893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.328874    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.329744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.329625    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:12.330337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.330318    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs\") pod \"a4bbf043-42ca-4359-92b7-fb8ec834da65\" (UID: \"a4bbf043-42ca-4359-92b7-fb8ec834da65\") "
Nov 03 03:34:12.330606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.330571    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:12.331037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.331009    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bbf043-42ca-4359-92b7-fb8ec834da65-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.331116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.331042    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.331116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.331058    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.331116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.331073    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.331116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.331086    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.383989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.383751    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:12.383989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.383896    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:34:12.392031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.391992    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "kube-api-access-p4q2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:12.394113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.392694    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy" pod "a4bbf043-42ca-4359-92b7-fb8ec834da65" (UID: "a4bbf043-42ca-4359-92b7-fb8ec834da65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:12.432888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.432838    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4q2r\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-kube-api-access-p4q2r\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.432888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.432870    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.432888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.432879    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:12.432888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:12.432887    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bbf043-42ca-4359-92b7-fb8ec834da65-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.082796 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.082762    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"
Nov 03 03:34:13.082980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.082789    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md" event={"ID":"a4bbf043-42ca-4359-92b7-fb8ec834da65","Type":"ContainerDied","Data":"9d82514614369f468e1def3c2e8c67ab8cbe5f5430bb6eeb69204e7c872557e2"}
Nov 03 03:34:13.082980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.082858    5515 scope.go:122] "RemoveContainer" containerID="9963df0cece920f5d612b8ae406e3e3d24c558845049c0a67f2168160ed048c0"
Nov 03 03:34:13.302372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.302300    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"]
Nov 03 03:34:13.353501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.353471    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-64c59d-openshift-default-7666bb5db7-d49md"]
Nov 03 03:34:13.375923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.375900    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:13.729703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.729679    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:34:13.753008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.752976    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753047    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753109    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753206    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753247    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753373 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753324    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:13.753428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753353    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753404    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753405    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:13.753611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753481    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753599    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo\") pod \"988a661b-4a24-4a07-8c85-953950bdb3f2\" (UID: \"988a661b-4a24-4a07-8c85-953950bdb3f2\") "
Nov 03 03:34:13.753735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753595    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:13.753735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753723    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:13.753955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.753928    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:13.754300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.754277    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.754384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.754306    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.754384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.754321    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/988a661b-4a24-4a07-8c85-953950bdb3f2-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.754384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.754336    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.754384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.754351    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.755400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.755376    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "kube-api-access-n844c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:13.755683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.755661    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:13.755751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.755731    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:13.755853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.755837    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo" pod "988a661b-4a24-4a07-8c85-953950bdb3f2" (UID: "988a661b-4a24-4a07-8c85-953950bdb3f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:34:13.863573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.855730    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.863573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.855773    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.863573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.855789    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n844c\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-kube-api-access-n844c\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:13.863573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:13.855803    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/988a661b-4a24-4a07-8c85-953950bdb3f2-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:14.092740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.092635    5515 generic.go:356] "Generic (PLEG): container finished" podID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerID="8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d" exitCode=0
Nov 03 03:34:14.092740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.092691    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"
Nov 03 03:34:14.092740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.092721    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" event={"ID":"988a661b-4a24-4a07-8c85-953950bdb3f2","Type":"ContainerDied","Data":"8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d"}
Nov 03 03:34:14.093029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.092782    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt" event={"ID":"988a661b-4a24-4a07-8c85-953950bdb3f2","Type":"ContainerDied","Data":"2d7f3e61824d01feae14d963f01d59de9cef865e482e5a90e14ee51c28095e89"}
Nov 03 03:34:14.093029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.092813    5515 scope.go:122] "RemoveContainer" containerID="8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d"
Nov 03 03:34:14.104217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.104201    5515 scope.go:122] "RemoveContainer" containerID="d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"
Nov 03 03:34:14.113189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.113174    5515 scope.go:122] "RemoveContainer" containerID="8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d"
Nov 03 03:34:14.113453 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:14.113437    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d\": container with ID starting with 8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d not found: ID does not exist" containerID="8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d"
Nov 03 03:34:14.113490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.113462    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d"} err="failed to get container status \"8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d\": rpc error: code = NotFound desc = could not find container \"8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d\": container with ID starting with 8f89aed901fa965be5517818119247d932de490cef2922b9d85b8deed503189d not found: ID does not exist"
Nov 03 03:34:14.113490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.113479    5515 scope.go:122] "RemoveContainer" containerID="d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"
Nov 03 03:34:14.113747 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:14.113728    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb\": container with ID starting with d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb not found: ID does not exist" containerID="d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"
Nov 03 03:34:14.113798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.113751    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb"} err="failed to get container status \"d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb\": rpc error: code = NotFound desc = could not find container \"d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb\": container with ID starting with d171f7ce372c845f1923a6e8185c2182c69184262402aa15d1aec98aa3af14fb not found: ID does not exist"
Nov 03 03:34:14.261233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.261194    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"]
Nov 03 03:34:14.264782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.264757    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9902a0c2-openshift-default-777895b7fc-6tdrt"]
Nov 03 03:34:14.336562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.336524    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:14.766328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.766279    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:14.874303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874269    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"]
Nov 03 03:34:14.874890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874874    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d05c116a-060b-483a-a9af-9e2b9c7d2148" containerName="maas-api"
Nov 03 03:34:14.874890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874891    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05c116a-060b-483a-a9af-9e2b9c7d2148" containerName="maas-api"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874906    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874911    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874930    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cbfaf6c4-1871-464a-8839-76c4100c7e79" containerName="payload-pre-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874936    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfaf6c4-1871-464a-8839-76c4100c7e79" containerName="payload-pre-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874952    5515 cpu_manager.go:414] "containerMap: removing container" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874957    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874969    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4d0f56d0-f2da-4506-bda0-9b971be15163" containerName="payload-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874973    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0f56d0-f2da-4506-bda0-9b971be15163" containerName="payload-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874985    5515 cpu_manager.go:414] "containerMap: removing container" podUID="87980c56-7904-4dcb-8297-36cfd18adc3a" containerName="payload-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874990    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87980c56-7904-4dcb-8297-36cfd18adc3a" containerName="payload-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.874997    5515 cpu_manager.go:414] "containerMap: removing container" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875002    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875008    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0ce923f5-be32-4b99-bcf7-1f0084f8c81a" containerName="payload-pre-processing"
Nov 03 03:34:14.875018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875013    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce923f5-be32-4b99-bcf7-1f0084f8c81a" containerName="payload-pre-processing"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875107    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05c116a-060b-483a-a9af-9e2b9c7d2148" containerName="maas-api"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875114    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875124    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfaf6c4-1871-464a-8839-76c4100c7e79" containerName="payload-pre-processing"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875132    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875138    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0f56d0-f2da-4506-bda0-9b971be15163" containerName="payload-processing"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875145    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="87980c56-7904-4dcb-8297-36cfd18adc3a" containerName="payload-processing"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875152    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce923f5-be32-4b99-bcf7-1f0084f8c81a" containerName="payload-pre-processing"
Nov 03 03:34:14.875416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.875159    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" containerName="istio-proxy"
Nov 03 03:34:14.880167 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.880140    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:14.979537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:14.977205    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f1c469-openshift-default-dockercfg-2dxm9\""
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.008845    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.008905    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5ql\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.008945    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.009012    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.009079    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.009112    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.009156    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.009267    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.009725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.009290    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.025265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.023521    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:15.032701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.030256    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="988a661b-4a24-4a07-8c85-953950bdb3f2" path="/var/lib/kubelet/pods/988a661b-4a24-4a07-8c85-953950bdb3f2/volumes"
Nov 03 03:34:15.032701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.031016    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" path="/var/lib/kubelet/pods/a4bbf043-42ca-4359-92b7-fb8ec834da65/volumes"
Nov 03 03:34:15.032701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.031454    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"]
Nov 03 03:34:15.110378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110149    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110194    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110235    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110271    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5ql\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110307    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110363    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110419    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110448    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.110892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.110488    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.111412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.111379    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.111733 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.111710    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.111918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.111902    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.112025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.111944    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.114932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.112361    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.115598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.115566    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.115898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.115876    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.163203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.163172    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"]
Nov 03 03:34:15.177084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.177056    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.177226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.177207    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5ql\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql\") pod \"e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") " pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.267068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.267028    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"]
Nov 03 03:34:15.282574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.282443    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:15.529397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.529362    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:15.644489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.644451    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"]
Nov 03 03:34:15.644833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.644805    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps" podUID="aadefec4-c215-4e51-93c0-489eee7b720c" containerName="maas-api" containerID="cri-o://5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393" gracePeriod=30
Nov 03 03:34:15.675522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.675494    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:15.715527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.715500    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"]
Nov 03 03:34:15.722648 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:15.722623    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c240ad0_f62f_4516_8824_c710d0e960c9.slice/crio-c642e944caf33bbb735af4158b1505bd10e3be10cb91d3e390151efe16e7da1a WatchSource:0}: Error finding container c642e944caf33bbb735af4158b1505bd10e3be10cb91d3e390151efe16e7da1a: Status 404 returned error can't find the container with id c642e944caf33bbb735af4158b1505bd10e3be10cb91d3e390151efe16e7da1a
Nov 03 03:34:15.752151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.752118    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:15.752276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.752200    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:15.752276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.752250    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:15.847059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:15.846875    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:16.003110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.002766    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:34:16.048600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.048534    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\") pod \"aadefec4-c215-4e51-93c0-489eee7b720c\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") "
Nov 03 03:34:16.048807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.048665    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca\") pod \"aadefec4-c215-4e51-93c0-489eee7b720c\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") "
Nov 03 03:34:16.048807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.048736    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs\") pod \"aadefec4-c215-4e51-93c0-489eee7b720c\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") "
Nov 03 03:34:16.049023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.048994    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca" pod "aadefec4-c215-4e51-93c0-489eee7b720c" (UID: "aadefec4-c215-4e51-93c0-489eee7b720c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:16.049278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.049258    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aadefec4-c215-4e51-93c0-489eee7b720c-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:16.050946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.050927    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs" pod "aadefec4-c215-4e51-93c0-489eee7b720c" (UID: "aadefec4-c215-4e51-93c0-489eee7b720c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:34:16.051016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.050949    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls" pod "aadefec4-c215-4e51-93c0-489eee7b720c" (UID: "aadefec4-c215-4e51-93c0-489eee7b720c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:34:16.120344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.120309    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" event={"ID":"5c240ad0-f62f-4516-8824-c710d0e960c9","Type":"ContainerStarted","Data":"fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"}
Nov 03 03:34:16.120533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.120365    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" event={"ID":"5c240ad0-f62f-4516-8824-c710d0e960c9","Type":"ContainerStarted","Data":"c642e944caf33bbb735af4158b1505bd10e3be10cb91d3e390151efe16e7da1a"}
Nov 03 03:34:16.122109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.122078    5515 generic.go:356] "Generic (PLEG): container finished" podID="aadefec4-c215-4e51-93c0-489eee7b720c" containerID="5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393" exitCode=0
Nov 03 03:34:16.122285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.122177    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps" event={"ID":"aadefec4-c215-4e51-93c0-489eee7b720c","Type":"ContainerDied","Data":"5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393"}
Nov 03 03:34:16.122285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.122220    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps" event={"ID":"aadefec4-c215-4e51-93c0-489eee7b720c","Type":"ContainerDied","Data":"16f3e57ff5ab3f6dd3c29faae39b08f2c0f7df2c104532784b82c194327e8676"}
Nov 03 03:34:16.122285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.122252    5515 scope.go:122] "RemoveContainer" containerID="5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393"
Nov 03 03:34:16.122447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.122378    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"
Nov 03 03:34:16.170279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.153083    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc\" (UniqueName: \"kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc\") pod \"aadefec4-c215-4e51-93c0-489eee7b720c\" (UID: \"aadefec4-c215-4e51-93c0-489eee7b720c\") "
Nov 03 03:34:16.170279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.169877    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc" pod "aadefec4-c215-4e51-93c0-489eee7b720c" (UID: "aadefec4-c215-4e51-93c0-489eee7b720c"). InnerVolumeSpecName "kube-api-access-m6lsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:16.175394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.171534    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:16.175394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.171587    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aadefec4-c215-4e51-93c0-489eee7b720c-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:16.175394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.171602    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6lsc\" (UniqueName: \"kubernetes.io/projected/aadefec4-c215-4e51-93c0-489eee7b720c-kube-api-access-m6lsc\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:16.181085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.181058    5515 scope.go:122] "RemoveContainer" containerID="5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393"
Nov 03 03:34:16.181529 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:16.181472    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393\": container with ID starting with 5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393 not found: ID does not exist" containerID="5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393"
Nov 03 03:34:16.182196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.182160    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393"} err="failed to get container status \"5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393\": rpc error: code = NotFound desc = could not find container \"5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393\": container with ID starting with 5e0ad7558a0b91459d6ac33d2d6ea25f912e4abc9268fd26dcbc8349d387e393 not found: ID does not exist"
Nov 03 03:34:16.247576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.247481    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podStartSLOduration=2.247462988 podStartE2EDuration="2.247462988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:16.245948686 +0000 UTC m=+1947.476841106" watchObservedRunningTime="2026-11-03 03:34:16.247462988 +0000 UTC m=+1947.478355406"
Nov 03 03:34:16.282871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.282837    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:16.284595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.284568    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 03:34:16.284745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.284621    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 03:34:16.544970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.544938    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"]
Nov 03 03:34:16.548928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.548900    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-687adc-dcd886f74-wqbps"]
Nov 03 03:34:16.896083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.896042    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadefec4-c215-4e51-93c0-489eee7b720c" path="/var/lib/kubelet/pods/aadefec4-c215-4e51-93c0-489eee7b720c/volumes"
Nov 03 03:34:16.931654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:16.931625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:17.349324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:17.349233    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:17.783584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:17.783528    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:17.975068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:17.975008    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:18.146889 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:18.146857    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:18.148090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:18.148066    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:34:19.031459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:19.031430    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:19.236941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:19.236660    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:19.455808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:19.455774    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:19.609605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:19.609541    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:19.837622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:19.837511    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:20.058579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.058517    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:20.163616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.163576    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8" podUID="ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" containerName="payload-processing" containerID="cri-o://164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77" gracePeriod=25
Nov 03 03:34:20.268377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.268327    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq" podUID="ccd80adf-3acf-4cf7-805f-777bdbc84bb6" containerName="payload-pre-processing" containerID="cri-o://70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688" gracePeriod=25
Nov 03 03:34:20.426629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.426558    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:34:20.449583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.448494    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg\" (UniqueName: \"kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg\") pod \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") "
Nov 03 03:34:20.449583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.448606    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume\") pod \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\" (UID: \"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018\") "
Nov 03 03:34:20.449583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.449120    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume" pod "ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" (UID: "ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:20.449583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.449277    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:20.451217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.451187    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg" pod "ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" (UID: "ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018"). InnerVolumeSpecName "kube-api-access-ptswg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:20.530708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.530672    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:34:20.549712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.549681    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t\" (UniqueName: \"kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t\") pod \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") "
Nov 03 03:34:20.549872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.549841    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume\") pod \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\" (UID: \"ccd80adf-3acf-4cf7-805f-777bdbc84bb6\") "
Nov 03 03:34:20.550106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.550090    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptswg\" (UniqueName: \"kubernetes.io/projected/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018-kube-api-access-ptswg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:20.550198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.550176    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume" pod "ccd80adf-3acf-4cf7-805f-777bdbc84bb6" (UID: "ccd80adf-3acf-4cf7-805f-777bdbc84bb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:20.551991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.551965    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t" pod "ccd80adf-3acf-4cf7-805f-777bdbc84bb6" (UID: "ccd80adf-3acf-4cf7-805f-777bdbc84bb6"). InnerVolumeSpecName "kube-api-access-hzj8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:20.651333 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.651295    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:20.651333 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:20.651328    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzj8t\" (UniqueName: \"kubernetes.io/projected/ccd80adf-3acf-4cf7-805f-777bdbc84bb6-kube-api-access-hzj8t\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:20.984269 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:20.984231    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:34:21.058177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.058139    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:21.168177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.168085    5515 generic.go:356] "Generic (PLEG): container finished" podID="ccd80adf-3acf-4cf7-805f-777bdbc84bb6" containerID="70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688" exitCode=0
Nov 03 03:34:21.168177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.168142    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"
Nov 03 03:34:21.168693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.168170    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq" event={"ID":"ccd80adf-3acf-4cf7-805f-777bdbc84bb6","Type":"ContainerDied","Data":"70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688"}
Nov 03 03:34:21.168693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.168224    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq" event={"ID":"ccd80adf-3acf-4cf7-805f-777bdbc84bb6","Type":"ContainerDied","Data":"d77e36520bd746897cfe30aaa907f8900449ee9aea7a2fcbecefa013f2b50c6b"}
Nov 03 03:34:21.168693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.168247    5515 scope.go:122] "RemoveContainer" containerID="70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688"
Nov 03 03:34:21.169766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.169746    5515 generic.go:356] "Generic (PLEG): container finished" podID="ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" containerID="164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77" exitCode=0
Nov 03 03:34:21.169871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.169788    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"
Nov 03 03:34:21.169871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.169836    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8" event={"ID":"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018","Type":"ContainerDied","Data":"164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77"}
Nov 03 03:34:21.169950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.169883    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8" event={"ID":"ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018","Type":"ContainerDied","Data":"1675b513ed9580a107a86211039f5f7b140a2cb8892142df4ca236e0418d941c"}
Nov 03 03:34:21.190397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.190378    5515 scope.go:122] "RemoveContainer" containerID="70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688"
Nov 03 03:34:21.190724 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:21.190706    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688\": container with ID starting with 70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688 not found: ID does not exist" containerID="70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688"
Nov 03 03:34:21.190799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.190731    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688"} err="failed to get container status \"70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688\": rpc error: code = NotFound desc = could not find container \"70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688\": container with ID starting with 70db3859726af56a2494e92dbe42c0e8bb4b29c056bd6d1c6bad1257f5724688 not found: ID does not exist"
Nov 03 03:34:21.190799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.190748    5515 scope.go:122] "RemoveContainer" containerID="164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77"
Nov 03 03:34:21.199864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.199832    5515 scope.go:122] "RemoveContainer" containerID="164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77"
Nov 03 03:34:21.200141 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:21.200120    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77\": container with ID starting with 164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77 not found: ID does not exist" containerID="164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77"
Nov 03 03:34:21.200199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.200148    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77"} err="failed to get container status \"164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77\": rpc error: code = NotFound desc = could not find container \"164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77\": container with ID starting with 164f1211d84fec51d2f78167a562ef6f3ee2767e5f5859836699f3610c66db77 not found: ID does not exist"
Nov 03 03:34:21.338677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.338644    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"]
Nov 03 03:34:21.389602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.389567    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-687adc-6cf8cf8c-9f2s8"]
Nov 03 03:34:21.397468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.397432    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:21.500160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.500084    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"]
Nov 03 03:34:21.507600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.507566    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-687adc-cf84d975f-nhvqq"]
Nov 03 03:34:21.543037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:21.543006    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:22.894130 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:22.894101    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd80adf-3acf-4cf7-805f-777bdbc84bb6" path="/var/lib/kubelet/pods/ccd80adf-3acf-4cf7-805f-777bdbc84bb6/volumes"
Nov 03 03:34:22.894534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:22.894519    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" path="/var/lib/kubelet/pods/ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018/volumes"
Nov 03 03:34:23.652515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:23.652482    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:24.152509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:24.152470    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:24.557993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:24.557890    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:24.948011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:24.947976    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:25.420421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.420387    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:25.654984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.654942    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:25.836332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836258    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"]
Nov 03 03:34:25.836860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836844    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ccd80adf-3acf-4cf7-805f-777bdbc84bb6" containerName="payload-pre-processing"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836861    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd80adf-3acf-4cf7-805f-777bdbc84bb6" containerName="payload-pre-processing"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836876    5515 cpu_manager.go:414] "containerMap: removing container" podUID="aadefec4-c215-4e51-93c0-489eee7b720c" containerName="maas-api"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836881    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadefec4-c215-4e51-93c0-489eee7b720c" containerName="maas-api"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836888    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836892    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836905    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" containerName="payload-processing"
Nov 03 03:34:25.836944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836910    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" containerName="payload-processing"
Nov 03 03:34:25.837156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836985    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd80adf-3acf-4cf7-805f-777bdbc84bb6" containerName="payload-pre-processing"
Nov 03 03:34:25.837156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.836995    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadefec4-c215-4e51-93c0-489eee7b720c" containerName="maas-api"
Nov 03 03:34:25.837156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.837002    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bbf043-42ca-4359-92b7-fb8ec834da65" containerName="istio-proxy"
Nov 03 03:34:25.837156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.837009    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1a6c1f-9790-4dfd-b1d0-ddddcbc66018" containerName="payload-processing"
Nov 03 03:34:25.841779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.841761    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:25.895138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.895100    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f1c469-dockercfg-qg2h4\""
Nov 03 03:34:25.961608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.953777    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f1c469\""
Nov 03 03:34:25.961608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.956295    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln57w\" (UniqueName: \"kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:25.961608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:25.956480    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:26.034753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.034711    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"]
Nov 03 03:34:26.040147 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.040122    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"]
Nov 03 03:34:26.066435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.066185    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:26.066435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.066336    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln57w\" (UniqueName: \"kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:26.067726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.067684    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:26.083522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.083496    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.087519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.087468    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"]
Nov 03 03:34:26.128437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.128407    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln57w\" (UniqueName: \"kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:26.155216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.155196    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:26.167723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.167700    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwp8p\" (UniqueName: \"kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p\") pod \"payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.167835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.167807    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.268955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.268920    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.269152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.269118    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwp8p\" (UniqueName: \"kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p\") pod \"payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.269725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.269705    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.331577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.331516    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwp8p\" (UniqueName: \"kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p\") pod \"payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.405391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.405352    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:26.532574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.532529    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"]
Nov 03 03:34:26.535131 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:26.535104    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed2befc_5860_43e2_b5b9_954c9b452da8.slice/crio-c04d45453e8a56a0b82c5f732a48db4985d1f4b7ff925a28c4845de68dad8124 WatchSource:0}: Error finding container c04d45453e8a56a0b82c5f732a48db4985d1f4b7ff925a28c4845de68dad8124: Status 404 returned error can't find the container with id c04d45453e8a56a0b82c5f732a48db4985d1f4b7ff925a28c4845de68dad8124
Nov 03 03:34:26.786978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:26.786954    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"]
Nov 03 03:34:26.788159 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:26.788133    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5a582b_1d3a_4a76_86dc_5d27b9eb6f16.slice/crio-36f1bb9b252e5808a8b7df344d8fb96093a24855d8598de0684e8555ad2ec69d WatchSource:0}: Error finding container 36f1bb9b252e5808a8b7df344d8fb96093a24855d8598de0684e8555ad2ec69d: Status 404 returned error can't find the container with id 36f1bb9b252e5808a8b7df344d8fb96093a24855d8598de0684e8555ad2ec69d
Nov 03 03:34:27.213616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.213575    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l" event={"ID":"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16","Type":"ContainerStarted","Data":"f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4"}
Nov 03 03:34:27.213830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.213634    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l" event={"ID":"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16","Type":"ContainerStarted","Data":"36f1bb9b252e5808a8b7df344d8fb96093a24855d8598de0684e8555ad2ec69d"}
Nov 03 03:34:27.213830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.213658    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:27.215133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.215110    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz" event={"ID":"0ed2befc-5860-43e2-b5b9-954c9b452da8","Type":"ContainerStarted","Data":"4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452"}
Nov 03 03:34:27.215213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.215149    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz" event={"ID":"0ed2befc-5860-43e2-b5b9-954c9b452da8","Type":"ContainerStarted","Data":"c04d45453e8a56a0b82c5f732a48db4985d1f4b7ff925a28c4845de68dad8124"}
Nov 03 03:34:27.215270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.215251    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:27.332704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.332652    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l" podStartSLOduration=2.332638435 podStartE2EDuration="2.332638435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:27.33097067 +0000 UTC m=+1958.561863088" watchObservedRunningTime="2026-11-03 03:34:27.332638435 +0000 UTC m=+1958.563530852"
Nov 03 03:34:27.435574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.435517    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:27.449069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.449020    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz" podStartSLOduration=2.449008304 podStartE2EDuration="2.449008304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:27.448482945 +0000 UTC m=+1958.679375363" watchObservedRunningTime="2026-11-03 03:34:27.449008304 +0000 UTC m=+1958.679900745"
Nov 03 03:34:27.587631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:27.586461    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:28.958328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:28.958295    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:29.469007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:29.468974    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"]
Nov 03 03:34:29.469007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:29.468984    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:29.469246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:29.469215    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy" containerID="cri-o://8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e" gracePeriod=30
Nov 03 03:34:29.849751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:29.849670    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:30.379669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:30.379629    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"]
Nov 03 03:34:30.437256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:30.437227    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:30.486475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:30.486439    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"]
Nov 03 03:34:30.739262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:30.739187    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:30.883273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:30.883246    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:31.066636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.066515    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"]
Nov 03 03:34:31.070806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.070785    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.164723    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.164861    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.164910    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.164961    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.165008    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlbj\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.165041    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.165100    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.165147    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.170574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.165179    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.195799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.194535    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:31.196134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.195712    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-160d39-openshift-default-dockercfg-z686q\""
Nov 03 03:34:31.197481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.197451    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"]
Nov 03 03:34:31.266372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.266332    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.266687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.266661    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.266915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.266890    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.267061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.267045    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlbj\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.267164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.267153    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.267298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.267285    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.267425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.267411    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.267579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.267565    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.267744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.267731    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.269999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.269433    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.269999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.269719    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.269999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.269957    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.270248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.270233    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.271631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.271326    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.290330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.287980    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.309806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.302579    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.331600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.331513    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlbj\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.332021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.331991    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token\") pod \"e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") " pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.385617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.385579    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:31.671539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.671247    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:31.862864 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:31.859817    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46f4684_5004_4dfe_aaaf_bb878fbf0e57.slice/crio-396926b7dbd03ba81aea7049078d065d2a9cf28e7f201c10faa65d760658d5cc WatchSource:0}: Error finding container 396926b7dbd03ba81aea7049078d065d2a9cf28e7f201c10faa65d760658d5cc: Status 404 returned error can't find the container with id 396926b7dbd03ba81aea7049078d065d2a9cf28e7f201c10faa65d760658d5cc
Nov 03 03:34:31.863320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.863269    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:31.863389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.863372    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:31.863436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.863417    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:31.863480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:31.863461    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"]
Nov 03 03:34:32.264960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.264884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:32.290641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.290607    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" event={"ID":"f46f4684-5004-4dfe-aaaf-bb878fbf0e57","Type":"ContainerStarted","Data":"efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"}
Nov 03 03:34:32.290816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.290668    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" event={"ID":"f46f4684-5004-4dfe-aaaf-bb878fbf0e57","Type":"ContainerStarted","Data":"396926b7dbd03ba81aea7049078d065d2a9cf28e7f201c10faa65d760658d5cc"}
Nov 03 03:34:32.385942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.385912    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:32.387380 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.387353    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 03 03:34:32.387524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.387404    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 03 03:34:32.419274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:32.419208    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podStartSLOduration=1.419189613 podStartE2EDuration="1.419189613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:32.417655079 +0000 UTC m=+1963.648547523" watchObservedRunningTime="2026-11-03 03:34:32.419189613 +0000 UTC m=+1963.650082035"
Nov 03 03:34:33.390937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:33.390910    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:34.305566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.305530    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:34.306591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.306568    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:34:34.742410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.742381    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:34:34.828527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828497    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828541    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828635    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828672    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828691    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828714    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828732    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828868    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.828966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.828916    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token\") pod \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\" (UID: \"3bfe2231-d5e3-4ed6-88c9-07483aae2c43\") "
Nov 03 03:34:34.829126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829007    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:34.829126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829026    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:34.829245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829157    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:34.829306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829248    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:34.829424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829402    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:34.829897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829875    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.829957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829906    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.829957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829920    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.829957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829949    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.830054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.829963    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.831025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.831007    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:34:34.831121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.831043    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "kube-api-access-gdmv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:34.831495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.831474    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:34:34.831495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.831494    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token" pod "3bfe2231-d5e3-4ed6-88c9-07483aae2c43" (UID: "3bfe2231-d5e3-4ed6-88c9-07483aae2c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:34.931710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.931674    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdmv7\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-kube-api-access-gdmv7\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.931710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.931702    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.931710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.931714    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:34.932002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:34.931722    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bfe2231-d5e3-4ed6-88c9-07483aae2c43-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:35.277840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.277805    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:35.313383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.313354    5515 generic.go:356] "Generic (PLEG): container finished" podID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerID="8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e" exitCode=0
Nov 03 03:34:35.313541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.313423    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"
Nov 03 03:34:35.313541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.313430    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" event={"ID":"3bfe2231-d5e3-4ed6-88c9-07483aae2c43","Type":"ContainerDied","Data":"8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e"}
Nov 03 03:34:35.313541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.313493    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv" event={"ID":"3bfe2231-d5e3-4ed6-88c9-07483aae2c43","Type":"ContainerDied","Data":"78d4ea80778cc9e5615cb956ed024182af4169939baa710e930e24304f32328c"}
Nov 03 03:34:35.313541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.313516    5515 scope.go:122] "RemoveContainer" containerID="8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e"
Nov 03 03:34:35.323404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.323386    5515 scope.go:122] "RemoveContainer" containerID="3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"
Nov 03 03:34:35.334938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.334890    5515 scope.go:122] "RemoveContainer" containerID="8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e"
Nov 03 03:34:35.335362 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:35.335335    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e\": container with ID starting with 8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e not found: ID does not exist" containerID="8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e"
Nov 03 03:34:35.335460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.335373    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e"} err="failed to get container status \"8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e\": rpc error: code = NotFound desc = could not find container \"8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e\": container with ID starting with 8b7259308808d63ee6c4c49895770eb67618100c8b77945141f8258be6db068e not found: ID does not exist"
Nov 03 03:34:35.335460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.335396    5515 scope.go:122] "RemoveContainer" containerID="3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"
Nov 03 03:34:35.335791 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:35.335771    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37\": container with ID starting with 3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37 not found: ID does not exist" containerID="3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"
Nov 03 03:34:35.335863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.335802    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37"} err="failed to get container status \"3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37\": rpc error: code = NotFound desc = could not find container \"3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37\": container with ID starting with 3e93a7aeaeba995aef2fdd0d5bd8199e913e7af3a01d9f1ba3a9f8b0ec02cb37 not found: ID does not exist"
Nov 03 03:34:35.379938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.379880    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl" podUID="9188f72b-439a-4dd8-ad15-06099a4f5614" containerName="payload-processing" containerID="cri-o://47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31" gracePeriod=25
Nov 03 03:34:35.431618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.431570    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:35.486923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.486887    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8" podUID="6fd91b6d-039a-4101-a8ec-48804f4c1cd2" containerName="payload-pre-processing" containerID="cri-o://74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1" gracePeriod=25
Nov 03 03:34:35.496120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.496091    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"]
Nov 03 03:34:35.503834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.503763    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-687adc-openshift-default-6c59bc89bf-hrghv"]
Nov 03 03:34:35.633575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.629147    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:35.791621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.791596    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:34:35.826353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.826332    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:34:35.850416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.850384    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr\" (UniqueName: \"kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr\") pod \"9188f72b-439a-4dd8-ad15-06099a4f5614\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") "
Nov 03 03:34:35.850582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.850512    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume\") pod \"9188f72b-439a-4dd8-ad15-06099a4f5614\" (UID: \"9188f72b-439a-4dd8-ad15-06099a4f5614\") "
Nov 03 03:34:35.850862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.850841    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume" pod "9188f72b-439a-4dd8-ad15-06099a4f5614" (UID: "9188f72b-439a-4dd8-ad15-06099a4f5614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:35.851031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.851015    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9188f72b-439a-4dd8-ad15-06099a4f5614-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:35.852608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.852590    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr" pod "9188f72b-439a-4dd8-ad15-06099a4f5614" (UID: "9188f72b-439a-4dd8-ad15-06099a4f5614"). InnerVolumeSpecName "kube-api-access-m5tfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:35.952009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.951937    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6\" (UniqueName: \"kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6\") pod \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") "
Nov 03 03:34:35.952009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.951978    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume\") pod \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\" (UID: \"6fd91b6d-039a-4101-a8ec-48804f4c1cd2\") "
Nov 03 03:34:35.952425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.952400    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume" pod "6fd91b6d-039a-4101-a8ec-48804f4c1cd2" (UID: "6fd91b6d-039a-4101-a8ec-48804f4c1cd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:35.952527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.952467    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5tfr\" (UniqueName: \"kubernetes.io/projected/9188f72b-439a-4dd8-ad15-06099a4f5614-kube-api-access-m5tfr\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:35.954227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:35.954206    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6" pod "6fd91b6d-039a-4101-a8ec-48804f4c1cd2" (UID: "6fd91b6d-039a-4101-a8ec-48804f4c1cd2"). InnerVolumeSpecName "kube-api-access-srbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:36.053778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.053742    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbm6\" (UniqueName: \"kubernetes.io/projected/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-kube-api-access-srbm6\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:36.053778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.053766    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd91b6d-039a-4101-a8ec-48804f4c1cd2-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:36.321951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.321859    5515 generic.go:356] "Generic (PLEG): container finished" podID="9188f72b-439a-4dd8-ad15-06099a4f5614" containerID="47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31" exitCode=0
Nov 03 03:34:36.321951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.321906    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"
Nov 03 03:34:36.321951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.321934    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl" event={"ID":"9188f72b-439a-4dd8-ad15-06099a4f5614","Type":"ContainerDied","Data":"47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31"}
Nov 03 03:34:36.322220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.321988    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl" event={"ID":"9188f72b-439a-4dd8-ad15-06099a4f5614","Type":"ContainerDied","Data":"32749fe08bad7f75f680aefb34f2a99eaf3298da71f2853fa1788963c49359de"}
Nov 03 03:34:36.322220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.322013    5515 scope.go:122] "RemoveContainer" containerID="47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31"
Nov 03 03:34:36.323536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.323513    5515 generic.go:356] "Generic (PLEG): container finished" podID="6fd91b6d-039a-4101-a8ec-48804f4c1cd2" containerID="74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1" exitCode=0
Nov 03 03:34:36.323804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.323592    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"
Nov 03 03:34:36.323804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.323603    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8" event={"ID":"6fd91b6d-039a-4101-a8ec-48804f4c1cd2","Type":"ContainerDied","Data":"74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1"}
Nov 03 03:34:36.323804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.323671    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8" event={"ID":"6fd91b6d-039a-4101-a8ec-48804f4c1cd2","Type":"ContainerDied","Data":"fc1c43a7786afdc9a36e54b9e498498e4f21ad01bdb9d783d628808baae74feb"}
Nov 03 03:34:36.333032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.333013    5515 scope.go:122] "RemoveContainer" containerID="47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31"
Nov 03 03:34:36.333366 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:36.333313    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31\": container with ID starting with 47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31 not found: ID does not exist" containerID="47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31"
Nov 03 03:34:36.333366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.333337    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31"} err="failed to get container status \"47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31\": rpc error: code = NotFound desc = could not find container \"47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31\": container with ID starting with 47cd810d3ebde15bf2df919e69d4644075daf945aa1891f071bc590db6014f31 not found: ID does not exist"
Nov 03 03:34:36.333366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.333361    5515 scope.go:122] "RemoveContainer" containerID="74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1"
Nov 03 03:34:36.342941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.342924    5515 scope.go:122] "RemoveContainer" containerID="74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1"
Nov 03 03:34:36.343197 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:36.343173    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1\": container with ID starting with 74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1 not found: ID does not exist" containerID="74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1"
Nov 03 03:34:36.343278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.343202    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1"} err="failed to get container status \"74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1\": rpc error: code = NotFound desc = could not find container \"74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1\": container with ID starting with 74388c4c814a7d86c44e518f78f0602999788139b3533bb636a697b59cc920b1 not found: ID does not exist"
Nov 03 03:34:36.491225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.491195    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"]
Nov 03 03:34:36.541174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.541138    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-211ed3-5ff8b48f5c-pjxrl"]
Nov 03 03:34:36.603822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.603747    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"]
Nov 03 03:34:36.607687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.607665    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-211ed3-65b55c5d55-dx2w8"]
Nov 03 03:34:36.893715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.893679    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" path="/var/lib/kubelet/pods/3bfe2231-d5e3-4ed6-88c9-07483aae2c43/volumes"
Nov 03 03:34:36.894268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.894247    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd91b6d-039a-4101-a8ec-48804f4c1cd2" path="/var/lib/kubelet/pods/6fd91b6d-039a-4101-a8ec-48804f4c1cd2/volumes"
Nov 03 03:34:36.894707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:36.894687    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9188f72b-439a-4dd8-ad15-06099a4f5614" path="/var/lib/kubelet/pods/9188f72b-439a-4dd8-ad15-06099a4f5614/volumes"
Nov 03 03:34:38.224609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.224541    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:34:38.224977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.224748    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:34:38.436108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.436080    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:38.583179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.583112    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:38.760573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.754981    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:38.851382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851309    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"]
Nov 03 03:34:38.851870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851856    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9188f72b-439a-4dd8-ad15-06099a4f5614" containerName="payload-processing"
Nov 03 03:34:38.851919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851872    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9188f72b-439a-4dd8-ad15-06099a4f5614" containerName="payload-processing"
Nov 03 03:34:38.851919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851900    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy"
Nov 03 03:34:38.851919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851905    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy"
Nov 03 03:34:38.851919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851915    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6fd91b6d-039a-4101-a8ec-48804f4c1cd2" containerName="payload-pre-processing"
Nov 03 03:34:38.852047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.851922    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd91b6d-039a-4101-a8ec-48804f4c1cd2" containerName="payload-pre-processing"
Nov 03 03:34:38.852047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.852014    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy"
Nov 03 03:34:38.852047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.852023    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9188f72b-439a-4dd8-ad15-06099a4f5614" containerName="payload-processing"
Nov 03 03:34:38.852047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.852035    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd91b6d-039a-4101-a8ec-48804f4c1cd2" containerName="payload-pre-processing"
Nov 03 03:34:38.856673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.856655    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:38.910675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.910647    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-160d39-dockercfg-prt6g\""
Nov 03 03:34:38.912967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.912940    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"]
Nov 03 03:34:38.965772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:38.965742    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-160d39\""
Nov 03 03:34:39.041749 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.041707    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"]
Nov 03 03:34:39.042878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.042848    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy"
Nov 03 03:34:39.043121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.043100    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy"
Nov 03 03:34:39.043590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.043569    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfe2231-d5e3-4ed6-88c9-07483aae2c43" containerName="istio-proxy"
Nov 03 03:34:39.050913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.048768    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.050913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.048831    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflcj\" (UniqueName: \"kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj\") pod \"payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.052360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.052339    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.075709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.075679    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"]
Nov 03 03:34:39.149708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.149679    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.149873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.149716    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wmr\" (UniqueName: \"kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr\") pod \"payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.149873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.149743    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflcj\" (UniqueName: \"kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj\") pod \"payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.149873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.149844    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.150349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.150331    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.211949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.211924    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflcj\" (UniqueName: \"kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj\") pod \"payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.251097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.251069    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wmr\" (UniqueName: \"kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr\") pod \"payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.251410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.251123    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.251813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.251793    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.313216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.313193    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wmr\" (UniqueName: \"kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr\") pod \"payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.404473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.404411    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:39.469964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.469919    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:39.780271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.780240    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"]
Nov 03 03:34:39.781838 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:39.781816    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e12cdc0_9324_46c1_b447_c10cc19401cc.slice/crio-68142758e5bbed251c92a9710980970e042cd2ff6a22379d286f1db53797b874 WatchSource:0}: Error finding container 68142758e5bbed251c92a9710980970e042cd2ff6a22379d286f1db53797b874: Status 404 returned error can't find the container with id 68142758e5bbed251c92a9710980970e042cd2ff6a22379d286f1db53797b874
Nov 03 03:34:39.858672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:39.858644    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"]
Nov 03 03:34:39.859353 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:39.859332    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464c2bed_831e_4193_bc77_27b480a6a28e.slice/crio-fbaea89f24e10cdf78538166d5b8210279f94808280c2c69fdfd747f68a91a86 WatchSource:0}: Error finding container fbaea89f24e10cdf78538166d5b8210279f94808280c2c69fdfd747f68a91a86: Status 404 returned error can't find the container with id fbaea89f24e10cdf78538166d5b8210279f94808280c2c69fdfd747f68a91a86
Nov 03 03:34:40.356567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.356515    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7" event={"ID":"464c2bed-831e-4193-bc77-27b480a6a28e","Type":"ContainerStarted","Data":"45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4"}
Nov 03 03:34:40.357030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.356613    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:40.357030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.356641    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7" event={"ID":"464c2bed-831e-4193-bc77-27b480a6a28e","Type":"ContainerStarted","Data":"fbaea89f24e10cdf78538166d5b8210279f94808280c2c69fdfd747f68a91a86"}
Nov 03 03:34:40.358199 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.358176    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx" event={"ID":"0e12cdc0-9324-46c1-b447-c10cc19401cc","Type":"ContainerStarted","Data":"02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47"}
Nov 03 03:34:40.358317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.358214    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx" event={"ID":"0e12cdc0-9324-46c1-b447-c10cc19401cc","Type":"ContainerStarted","Data":"68142758e5bbed251c92a9710980970e042cd2ff6a22379d286f1db53797b874"}
Nov 03 03:34:40.358317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.358281    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:40.479523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.479460    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7" podStartSLOduration=2.4794422369999998 podStartE2EDuration="2.479442237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:40.474233258 +0000 UTC m=+1971.705125678" watchObservedRunningTime="2026-11-03 03:34:40.479442237 +0000 UTC m=+1971.710334656"
Nov 03 03:34:40.528136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.528104    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:40.612137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:40.599232    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx" podStartSLOduration=2.599209119 podStartE2EDuration="2.599209119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:40.595956479 +0000 UTC m=+1971.826848892" watchObservedRunningTime="2026-11-03 03:34:40.599209119 +0000 UTC m=+1971.830101538"
Nov 03 03:34:41.326848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:41.326819    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:43.163899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:43.163864    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:34:45.430866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:45.430835    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:48.774771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:48.774740    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:49.657947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:49.657917    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:50.586018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:50.585938    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:50.736679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:50.733114    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:50.985232 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:50.985193    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:34:51.152060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.152016    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"]
Nov 03 03:34:51.158196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.158170    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.205451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.205406    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.207040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.207011    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.207337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.207315    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.207581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.207543    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.207747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.207730    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.211571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.207857    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.232573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.211814    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.232861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.232833    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.233092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.233074    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwx48\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.233319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.211500    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b6ef2c-openshift-default-dockercfg-nvn6m\""
Nov 03 03:34:51.280623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.273311    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:51.295582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.295191    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"]
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.340807    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.340869    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.340969    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwx48\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341021    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341047    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341098    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341201    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341228    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341259    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.341924    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.342727    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.343492    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.344193    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.344419    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.364570    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.388475    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.394620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.392850    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:34:51.403826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.398748    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:34:51.432598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.432566    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.433075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.433055    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwx48\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48\") pod \"e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") " pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.506229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.506193    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:51.700438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.700408    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:51.974366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.974330    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"]
Nov 03 03:34:51.986057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.986031    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"]
Nov 03 03:34:51.987978 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:34:51.987956    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcad0ae_8c13_431c_a3f1_ac2733e2b488.slice/crio-ef839ab0eb81395de7e1b4984b4e2c3a52bf3ee3eec9e895aee5ab9e43350bf8 WatchSource:0}: Error finding container ef839ab0eb81395de7e1b4984b4e2c3a52bf3ee3eec9e895aee5ab9e43350bf8: Status 404 returned error can't find the container with id ef839ab0eb81395de7e1b4984b4e2c3a52bf3ee3eec9e895aee5ab9e43350bf8
Nov 03 03:34:51.990469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.990435    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:51.990602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.990510    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:51.990602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:51.990579    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:34:52.085940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.085899    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"]
Nov 03 03:34:52.339894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.339801    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:52.481992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.481783    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" event={"ID":"9dcad0ae-8c13-431c-a3f1-ac2733e2b488","Type":"ContainerStarted","Data":"c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"}
Nov 03 03:34:52.482256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.482235    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" event={"ID":"9dcad0ae-8c13-431c-a3f1-ac2733e2b488","Type":"ContainerStarted","Data":"ef839ab0eb81395de7e1b4984b4e2c3a52bf3ee3eec9e895aee5ab9e43350bf8"}
Nov 03 03:34:52.494118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.494081    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:52.512612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.509289    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:52.608218 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:52.608161    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podStartSLOduration=1.6081428180000001 podStartE2EDuration="1.608142818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:34:52.606825716 +0000 UTC m=+1983.837718137" watchObservedRunningTime="2026-11-03 03:34:52.608142818 +0000 UTC m=+1983.839035237"
Nov 03 03:34:53.579291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:53.579252    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:54.530652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:54.530611    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:54.531682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:54.531663    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:34:56.263701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:56.263664    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:56.409222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:56.409187    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:56.649737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:56.649710    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:34:56.975285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:56.975194    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6" podUID="ee494ea4-9f1c-4050-9449-f4918beb65d1" containerName="payload-processing" containerID="cri-o://d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe" gracePeriod=25
Nov 03 03:34:57.086965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.086928    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl" podUID="6feba1c5-8767-405f-9ce5-074146aeadf0" containerName="payload-pre-processing" containerID="cri-o://eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729" gracePeriod=25
Nov 03 03:34:57.239719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.239652    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:34:57.336470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.336446    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume\") pod \"ee494ea4-9f1c-4050-9449-f4918beb65d1\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") "
Nov 03 03:34:57.336807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.336718    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg\" (UniqueName: \"kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg\") pod \"ee494ea4-9f1c-4050-9449-f4918beb65d1\" (UID: \"ee494ea4-9f1c-4050-9449-f4918beb65d1\") "
Nov 03 03:34:57.336862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.336841    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume" pod "ee494ea4-9f1c-4050-9449-f4918beb65d1" (UID: "ee494ea4-9f1c-4050-9449-f4918beb65d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:57.337188 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.337169    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee494ea4-9f1c-4050-9449-f4918beb65d1-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:57.339111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.339087    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg" pod "ee494ea4-9f1c-4050-9449-f4918beb65d1" (UID: "ee494ea4-9f1c-4050-9449-f4918beb65d1"). InnerVolumeSpecName "kube-api-access-7vvzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:57.354122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.354101    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:34:57.437980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.437951    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq\" (UniqueName: \"kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq\") pod \"6feba1c5-8767-405f-9ce5-074146aeadf0\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") "
Nov 03 03:34:57.438140 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.438020    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume\") pod \"6feba1c5-8767-405f-9ce5-074146aeadf0\" (UID: \"6feba1c5-8767-405f-9ce5-074146aeadf0\") "
Nov 03 03:34:57.438387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.438373    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvzg\" (UniqueName: \"kubernetes.io/projected/ee494ea4-9f1c-4050-9449-f4918beb65d1-kube-api-access-7vvzg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:57.438456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.438380    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume" pod "6feba1c5-8767-405f-9ce5-074146aeadf0" (UID: "6feba1c5-8767-405f-9ce5-074146aeadf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:34:57.440229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.440210    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq" pod "6feba1c5-8767-405f-9ce5-074146aeadf0" (UID: "6feba1c5-8767-405f-9ce5-074146aeadf0"). InnerVolumeSpecName "kube-api-access-tv9jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:34:57.540024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.539938    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv9jq\" (UniqueName: \"kubernetes.io/projected/6feba1c5-8767-405f-9ce5-074146aeadf0-kube-api-access-tv9jq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:57.540024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.539968    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6feba1c5-8767-405f-9ce5-074146aeadf0-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:34:57.549874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.549841    5515 generic.go:356] "Generic (PLEG): container finished" podID="6feba1c5-8767-405f-9ce5-074146aeadf0" containerID="eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729" exitCode=0
Nov 03 03:34:57.550028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.549912    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"
Nov 03 03:34:57.550028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.549928    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl" event={"ID":"6feba1c5-8767-405f-9ce5-074146aeadf0","Type":"ContainerDied","Data":"eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729"}
Nov 03 03:34:57.550028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.550007    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl" event={"ID":"6feba1c5-8767-405f-9ce5-074146aeadf0","Type":"ContainerDied","Data":"c35ada27cfc286072ef76314dc0e595f34f6234706e581c6a735ff3a247c107f"}
Nov 03 03:34:57.550171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.550038    5515 scope.go:122] "RemoveContainer" containerID="eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729"
Nov 03 03:34:57.551439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.551410    5515 generic.go:356] "Generic (PLEG): container finished" podID="ee494ea4-9f1c-4050-9449-f4918beb65d1" containerID="d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe" exitCode=0
Nov 03 03:34:57.551593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.551489    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6" event={"ID":"ee494ea4-9f1c-4050-9449-f4918beb65d1","Type":"ContainerDied","Data":"d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe"}
Nov 03 03:34:57.551593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.551537    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6" event={"ID":"ee494ea4-9f1c-4050-9449-f4918beb65d1","Type":"ContainerDied","Data":"3fe0e85eb8eb624b11922d1add0fcbf9ae90504209e7357b7ef3b21a8a4e2de4"}
Nov 03 03:34:57.551686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.551661    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"
Nov 03 03:34:57.562007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.561987    5515 scope.go:122] "RemoveContainer" containerID="eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729"
Nov 03 03:34:57.562293 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:57.562276    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729\": container with ID starting with eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729 not found: ID does not exist" containerID="eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729"
Nov 03 03:34:57.562372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.562299    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729"} err="failed to get container status \"eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729\": rpc error: code = NotFound desc = could not find container \"eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729\": container with ID starting with eb744f2e7d547f928cd513f5824a6d108833bebd3ea7819fb23d315bf19e3729 not found: ID does not exist"
Nov 03 03:34:57.562372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.562315    5515 scope.go:122] "RemoveContainer" containerID="d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe"
Nov 03 03:34:57.570867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.570849    5515 scope.go:122] "RemoveContainer" containerID="d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe"
Nov 03 03:34:57.571143 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:34:57.571124    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe\": container with ID starting with d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe not found: ID does not exist" containerID="d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe"
Nov 03 03:34:57.571196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.571150    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe"} err="failed to get container status \"d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe\": rpc error: code = NotFound desc = could not find container \"d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe\": container with ID starting with d03ccd886cdcb273a0fa27d70b3a2b6e3db26eeaf671d9ebdd3cda6b6e74aafe not found: ID does not exist"
Nov 03 03:34:57.716663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.716634    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"]
Nov 03 03:34:57.721506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.721479    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-59d699df8f-btmz6"]
Nov 03 03:34:57.832847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.832771    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"]
Nov 03 03:34:57.838159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:57.838132    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-689978c887-252kl"]
Nov 03 03:34:58.896143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:58.896112    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6feba1c5-8767-405f-9ce5-074146aeadf0" path="/var/lib/kubelet/pods/6feba1c5-8767-405f-9ce5-074146aeadf0/volumes"
Nov 03 03:34:58.896627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:58.896498    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee494ea4-9f1c-4050-9449-f4918beb65d1" path="/var/lib/kubelet/pods/ee494ea4-9f1c-4050-9449-f4918beb65d1/volumes"
Nov 03 03:34:58.986526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:34:58.986494    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:00.036465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.036431    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:00.186480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.186454    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:00.360630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.357476    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:00.464214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464180    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"]
Nov 03 03:35:00.464857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464835    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6feba1c5-8767-405f-9ce5-074146aeadf0" containerName="payload-pre-processing"
Nov 03 03:35:00.464857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464858    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6feba1c5-8767-405f-9ce5-074146aeadf0" containerName="payload-pre-processing"
Nov 03 03:35:00.465001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464881    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ee494ea4-9f1c-4050-9449-f4918beb65d1" containerName="payload-processing"
Nov 03 03:35:00.465001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464886    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee494ea4-9f1c-4050-9449-f4918beb65d1" containerName="payload-processing"
Nov 03 03:35:00.465001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464977    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6feba1c5-8767-405f-9ce5-074146aeadf0" containerName="payload-pre-processing"
Nov 03 03:35:00.465001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.464989    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee494ea4-9f1c-4050-9449-f4918beb65d1" containerName="payload-processing"
Nov 03 03:35:00.469612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.469595    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.526197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.526171    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"]
Nov 03 03:35:00.573158 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.573131    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6ef2c\""
Nov 03 03:35:00.575770 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.575748    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.575895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.575880    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9h85\" (UniqueName: \"kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.608105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.608076    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"]
Nov 03 03:35:00.608723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.608702    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:00.612945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.612897    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:00.681580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.671484    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6ef2c-dockercfg-5zwtl\""
Nov 03 03:35:00.681580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.679663    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.681580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.679812    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9h85\" (UniqueName: \"kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.681580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.681074    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.703064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.703025    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"]
Nov 03 03:35:00.765947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.765907    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9h85\" (UniqueName: \"kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85\") pod \"payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.791093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.781437    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87vt\" (UniqueName: \"kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt\") pod \"payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:00.791700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.791639    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:00.797621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.792302    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:00.905621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.901352    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:00.905621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.898541    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:00.905621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.902428    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87vt\" (UniqueName: \"kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt\") pod \"payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:00.967619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:00.967587    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87vt\" (UniqueName: \"kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt\") pod \"payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:01.226513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.226436    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:01.289583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.289538    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"]
Nov 03 03:35:01.295822 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:01.294787    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095f5fb8_5926_49c4_91da_d92ed249965c.slice/crio-00bc19341dddcd246d39f6b2960cc45a1d8aac8338f4345231c31987865c0239 WatchSource:0}: Error finding container 00bc19341dddcd246d39f6b2960cc45a1d8aac8338f4345231c31987865c0239: Status 404 returned error can't find the container with id 00bc19341dddcd246d39f6b2960cc45a1d8aac8338f4345231c31987865c0239
Nov 03 03:35:01.580493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.580409    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd" event={"ID":"095f5fb8-5926-49c4-91da-d92ed249965c","Type":"ContainerStarted","Data":"f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319"}
Nov 03 03:35:01.580651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.580492    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd" event={"ID":"095f5fb8-5926-49c4-91da-d92ed249965c","Type":"ContainerStarted","Data":"00bc19341dddcd246d39f6b2960cc45a1d8aac8338f4345231c31987865c0239"}
Nov 03 03:35:01.580651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.580636    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:01.607082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.607058    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"]
Nov 03 03:35:01.608150 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:01.608123    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e50ace_b076_4aea_bee9_1f2952628c29.slice/crio-435af77f6379f2a6d3ca8610ab57ce58f81f3bd4851f509bda4a6ad5ca499eb3 WatchSource:0}: Error finding container 435af77f6379f2a6d3ca8610ab57ce58f81f3bd4851f509bda4a6ad5ca499eb3: Status 404 returned error can't find the container with id 435af77f6379f2a6d3ca8610ab57ce58f81f3bd4851f509bda4a6ad5ca499eb3
Nov 03 03:35:01.698772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.698723    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd" podStartSLOduration=1.69870783 podStartE2EDuration="1.69870783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:01.69822308 +0000 UTC m=+1992.929115510" watchObservedRunningTime="2026-11-03 03:35:01.69870783 +0000 UTC m=+1992.929600247"
Nov 03 03:35:01.773414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.773380    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"]
Nov 03 03:35:01.777615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.777596    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:01.837661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.836353    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4a5156-dockercfg-l7trw\""
Nov 03 03:35:01.837661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.836480    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4a5156\""
Nov 03 03:35:01.844574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.842536    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"]
Nov 03 03:35:01.926140 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.926099    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxfmj\" (UniqueName: \"kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:01.926342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:01.926319    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:02.027411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.027374    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:02.027634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.027526    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxfmj\" (UniqueName: \"kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:02.028209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.028183    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:02.107169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.107103    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxfmj\" (UniqueName: \"kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj\") pod \"payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:02.395418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.395328    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:02.503891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.503853    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"]
Nov 03 03:35:02.509130 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.509110    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.532212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.532182    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c724\" (UniqueName: \"kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724\") pod \"payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.532361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.532327    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.565918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.565887    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"]
Nov 03 03:35:02.588809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.588775    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" event={"ID":"53e50ace-b076-4aea-bee9-1f2952628c29","Type":"ContainerStarted","Data":"1ec8f6f2ae7ba357c17081a46e6cee2257986477e105cd23af8ea43d2c890f67"}
Nov 03 03:35:02.588981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.588836    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" event={"ID":"53e50ace-b076-4aea-bee9-1f2952628c29","Type":"ContainerStarted","Data":"435af77f6379f2a6d3ca8610ab57ce58f81f3bd4851f509bda4a6ad5ca499eb3"}
Nov 03 03:35:02.588981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.588908    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:02.633323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.633292    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.633571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.633462    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c724\" (UniqueName: \"kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724\") pod \"payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.634067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.634045    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.695599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.695510    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c724\" (UniqueName: \"kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724\") pod \"payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:02.744107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.744050    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" podStartSLOduration=2.744037279 podStartE2EDuration="2.744037279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:02.743829774 +0000 UTC m=+1993.974722192" watchObservedRunningTime="2026-11-03 03:35:02.744037279 +0000 UTC m=+1993.974929697"
Nov 03 03:35:02.774672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.774643    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"]
Nov 03 03:35:02.775324 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:02.775298    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5013180_60c8_4188_9b82_3d00880d9d0f.slice/crio-b2e903303ecd37501f49a8181c9f765dde6c4768ab31601829f21167ab6444f7 WatchSource:0}: Error finding container b2e903303ecd37501f49a8181c9f765dde6c4768ab31601829f21167ab6444f7: Status 404 returned error can't find the container with id b2e903303ecd37501f49a8181c9f765dde6c4768ab31601829f21167ab6444f7
Nov 03 03:35:02.823389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:02.823352    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:03.273314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:03.273283    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"]
Nov 03 03:35:03.275019 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:03.274989    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5c93be_ba31_4cce_bbf1_11603e2eece4.slice/crio-1ae01a8da7507f150838bdac36ef364bf3b9f3cd427768091ede209ebf8e9bbf WatchSource:0}: Error finding container 1ae01a8da7507f150838bdac36ef364bf3b9f3cd427768091ede209ebf8e9bbf: Status 404 returned error can't find the container with id 1ae01a8da7507f150838bdac36ef364bf3b9f3cd427768091ede209ebf8e9bbf
Nov 03 03:35:03.415911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:03.415884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:03.600076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:03.599973    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874" event={"ID":"a5013180-60c8-4188-9b82-3d00880d9d0f","Type":"ContainerStarted","Data":"b2e903303ecd37501f49a8181c9f765dde6c4768ab31601829f21167ab6444f7"}
Nov 03 03:35:03.603079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:03.603043    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc" event={"ID":"6d5c93be-ba31-4cce-bbf1-11603e2eece4","Type":"ContainerStarted","Data":"1ae01a8da7507f150838bdac36ef364bf3b9f3cd427768091ede209ebf8e9bbf"}
Nov 03 03:35:05.622481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:05.622444    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874" event={"ID":"a5013180-60c8-4188-9b82-3d00880d9d0f","Type":"ContainerStarted","Data":"2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0"}
Nov 03 03:35:05.622940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:05.622502    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:05.624079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:05.624048    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc" event={"ID":"6d5c93be-ba31-4cce-bbf1-11603e2eece4","Type":"ContainerStarted","Data":"b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623"}
Nov 03 03:35:05.624156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:05.624108    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:05.741305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:05.741202    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874" podStartSLOduration=2.33995177 podStartE2EDuration="4.741189888s" totalImagesPullingTime="2.401238118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:05.740577288 +0000 UTC m=+1996.971469890" watchObservedRunningTime="2026-11-03 03:35:05.741189888 +0000 UTC m=+1996.972082306"
Nov 03 03:35:05.899758 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:05.899694    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc" podStartSLOduration=1.731654518 podStartE2EDuration="3.899669661s" totalImagesPullingTime="2.168015143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:05.887124598 +0000 UTC m=+1997.118017016" watchObservedRunningTime="2026-11-03 03:35:05.899669661 +0000 UTC m=+1997.130562084"
Nov 03 03:35:07.972881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:07.972844    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"]
Nov 03 03:35:07.977249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:07.977224    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.014674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.014631    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.014848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.014783    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.014848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.014818    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.014928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.014889    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsjp\" (UniqueName: \"kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.032777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.032742    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f1c469-serving-cert\""
Nov 03 03:35:08.087967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.087930    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"]
Nov 03 03:35:08.116181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.116131    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.116354 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.116275    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.116354 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.116308    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.116472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.116373    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsjp\" (UniqueName: \"kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.116529 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:08.116494    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-serving-cert: secret "maas-api-e2e-ipp-b-f1c469-serving-cert" not found
Nov 03 03:35:08.116626 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:08.116602    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls podName:42629e00-25a1-4c3a-ad4d-ececc0b29834 nodeName:}" failed. No retries permitted until 2026-11-03 03:35:08.616576804 +0000 UTC m=+1999.847469215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls") pod "maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" (UID: "42629e00-25a1-4c3a-ad4d-ececc0b29834") : secret "maas-api-e2e-ipp-b-f1c469-serving-cert" not found
Nov 03 03:35:08.117138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.117109    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.119024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.119002    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.175358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.175318    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:08.180934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.180892    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsjp\" (UniqueName: \"kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.332497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.332411    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:08.478950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.478916    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:08.622433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.622406    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.625008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.624981    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:08.894439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:08.894358    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:09.371690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:09.371660    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"]
Nov 03 03:35:09.372651 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:09.372622    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42629e00_25a1_4c3a_ad4d_ececc0b29834.slice/crio-28ff49a4e496bf02f7caf74db9f1780bcf7a07c8f49f4156be4ac9f7d527ffff WatchSource:0}: Error finding container 28ff49a4e496bf02f7caf74db9f1780bcf7a07c8f49f4156be4ac9f7d527ffff: Status 404 returned error can't find the container with id 28ff49a4e496bf02f7caf74db9f1780bcf7a07c8f49f4156be4ac9f7d527ffff
Nov 03 03:35:09.649592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:09.649493    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" event={"ID":"42629e00-25a1-4c3a-ad4d-ececc0b29834","Type":"ContainerStarted","Data":"28ff49a4e496bf02f7caf74db9f1780bcf7a07c8f49f4156be4ac9f7d527ffff"}
Nov 03 03:35:10.657627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:10.657595    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" event={"ID":"42629e00-25a1-4c3a-ad4d-ececc0b29834","Type":"ContainerStarted","Data":"52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593"}
Nov 03 03:35:10.658036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:10.657733    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:10.778892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:10.778830    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" podStartSLOduration=2.8294065010000002 podStartE2EDuration="3.778815077s" totalImagesPullingTime="949.408576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:10.776909361 +0000 UTC m=+2002.007801789" watchObservedRunningTime="2026-11-03 03:35:10.778815077 +0000 UTC m=+2002.009707494"
Nov 03 03:35:12.592981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:12.592951    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:35:13.185045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:13.185004    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:13.605848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:13.605761    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:35:16.633202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:16.633167    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:35:16.633656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:16.633240    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:35:16.670732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:16.670704    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:35:17.497574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:17.497527    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:19.294906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:19.294870    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:20.946517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:20.946482    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:20.988031 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:20.987995    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:35:21.094247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:21.094209    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:22.658803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:22.658771    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:23.209667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:23.209639    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"]
Nov 03 03:35:23.209863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:23.209833    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" containerID="cri-o://020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a" gracePeriod=30
Nov 03 03:35:23.211430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:23.211404    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:24.171654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.171621    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"]
Nov 03 03:35:24.176009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.175949    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.178865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.178842    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:24.185240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.185216    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.185315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.185258    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.185447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.185419    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4ns\" (UniqueName: \"kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.185631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.185615    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.230324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.230291    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-160d39-serving-cert\""
Nov 03 03:35:24.283951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.283920    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"]
Nov 03 03:35:24.286448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.286418    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4ns\" (UniqueName: \"kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.286621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.286604    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.286712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.286698    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.286780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.286747    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.286901 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:24.286879    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-serving-cert: secret "maas-api-e2e-ipp-a-160d39-serving-cert" not found
Nov 03 03:35:24.287028 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:24.286951    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls podName:04fb8e4e-456f-4f30-b902-81444f1030be nodeName:}" failed. No retries permitted until 2026-11-03 03:35:24.786930377 +0000 UTC m=+2016.017822792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls") pod "maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" (UID: "04fb8e4e-456f-4f30-b902-81444f1030be") : secret "maas-api-e2e-ipp-a-160d39-serving-cert" not found
Nov 03 03:35:24.287434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.287405    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.289494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.289472    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.349039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.349010    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4ns\" (UniqueName: \"kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.451800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.451710    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:24.615349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.614893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:24.813327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.812592    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:24.816462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:24.816434    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\") pod \"maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:25.090255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.090169    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:25.191468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.191398    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:25.403980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.403944    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:25.498787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.498759    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"]
Nov 03 03:35:25.499712 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:25.499681    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fb8e4e_456f_4f30_b902_81444f1030be.slice/crio-28c5feac7b4cecb9a8200e56558c7e0e15a2fa38d89252835f4d4cc602a43002 WatchSource:0}: Error finding container 28c5feac7b4cecb9a8200e56558c7e0e15a2fa38d89252835f4d4cc602a43002: Status 404 returned error can't find the container with id 28c5feac7b4cecb9a8200e56558c7e0e15a2fa38d89252835f4d4cc602a43002
Nov 03 03:35:25.641498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.641458    5515 patch_prober.go:29] interesting pod/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:35:25.641687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.641529    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:35:25.786499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:25.786417    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" event={"ID":"04fb8e4e-456f-4f30-b902-81444f1030be","Type":"ContainerStarted","Data":"28c5feac7b4cecb9a8200e56558c7e0e15a2fa38d89252835f4d4cc602a43002"}
Nov 03 03:35:26.801282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:26.801241    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" event={"ID":"04fb8e4e-456f-4f30-b902-81444f1030be","Type":"ContainerStarted","Data":"0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196"}
Nov 03 03:35:26.801761 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:26.801298    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:26.920774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:26.920727    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" podStartSLOduration=2.1059267249999998 podStartE2EDuration="2.920712671s" totalImagesPullingTime="814.785946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:26.920196001 +0000 UTC m=+2018.151088398" watchObservedRunningTime="2026-11-03 03:35:26.920712671 +0000 UTC m=+2018.151605089"
Nov 03 03:35:28.501181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.501156    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:35:28.568057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568027    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568109    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568142    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568177    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568214    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568254    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568293    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:35:28.568467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568300    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568448    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568514    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket\") pod \"7cb11c27-bf88-4249-8603-f41209cc645b\" (UID: \"7cb11c27-bf88-4249-8603-f41209cc645b\") "
Nov 03 03:35:28.568716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568524    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:35:28.568716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568630    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:35:28.568867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568726    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:35:28.568924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.568860    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:35:28.569331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.569307    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.569419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.569335    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb11c27-bf88-4249-8603-f41209cc645b-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.569419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.569350    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.569419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.569367    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.569419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.569383    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.570731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.570699    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:35:28.570731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.570711    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:35:28.570856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.570805    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "kube-api-access-2r9tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:35:28.570856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.570818    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token" pod "7cb11c27-bf88-4249-8603-f41209cc645b" (UID: "7cb11c27-bf88-4249-8603-f41209cc645b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:35:28.670441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.670404    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb11c27-bf88-4249-8603-f41209cc645b-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.670441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.670441    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.670671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.670459    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9tm\" (UniqueName: \"kubernetes.io/projected/7cb11c27-bf88-4249-8603-f41209cc645b-kube-api-access-2r9tm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.670671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.670475    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb11c27-bf88-4249-8603-f41209cc645b-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:35:28.815669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.815580    5515 generic.go:356] "Generic (PLEG): container finished" podID="7cb11c27-bf88-4249-8603-f41209cc645b" containerID="020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a" exitCode=0
Nov 03 03:35:28.815669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.815642    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"
Nov 03 03:35:28.815899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.815660    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" event={"ID":"7cb11c27-bf88-4249-8603-f41209cc645b","Type":"ContainerDied","Data":"020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a"}
Nov 03 03:35:28.815899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.815717    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694" event={"ID":"7cb11c27-bf88-4249-8603-f41209cc645b","Type":"ContainerDied","Data":"e4e62258c3b463941ced565198e62146c6b96e43aaf3fc251746d7da496d068e"}
Nov 03 03:35:28.815899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.815739    5515 scope.go:122] "RemoveContainer" containerID="020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a"
Nov 03 03:35:28.826941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.826923    5515 scope.go:122] "RemoveContainer" containerID="f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"
Nov 03 03:35:28.836324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.836298    5515 scope.go:122] "RemoveContainer" containerID="020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a"
Nov 03 03:35:28.836628 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:28.836608    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a\": container with ID starting with 020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a not found: ID does not exist" containerID="020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a"
Nov 03 03:35:28.836707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.836636    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a"} err="failed to get container status \"020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a\": rpc error: code = NotFound desc = could not find container \"020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a\": container with ID starting with 020920109fb4b7fbe8ad4dd1157ae25f93516740333f5e9f349294f89a3bcf3a not found: ID does not exist"
Nov 03 03:35:28.836707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.836653    5515 scope.go:122] "RemoveContainer" containerID="f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"
Nov 03 03:35:28.836895 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:28.836875    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938\": container with ID starting with f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938 not found: ID does not exist" containerID="f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"
Nov 03 03:35:28.836935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.836904    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938"} err="failed to get container status \"f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938\": rpc error: code = NotFound desc = could not find container \"f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938\": container with ID starting with f0622cf5fc483d0b92aa30bd7b2399e8566d7514660cb782ff9089bd62309938 not found: ID does not exist"
Nov 03 03:35:28.984444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.984396    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"]
Nov 03 03:35:28.990374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:28.990350    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-211ed3-openshift-default-56fdb7876c-n2694"]
Nov 03 03:35:29.871142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:29.871111    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:30.265818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:30.265735    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:30.449343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:30.449314    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:30.597394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:30.597306    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:30.898453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:30.898421    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" path="/var/lib/kubelet/pods/7cb11c27-bf88-4249-8603-f41209cc645b/volumes"
Nov 03 03:35:32.813906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:32.813875    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:35:33.488610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:33.488582    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:33.671831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:33.671795    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:34.224056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224027    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"]
Nov 03 03:35:34.224672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224657    5515 cpu_manager.go:414] "containerMap: removing container" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy"
Nov 03 03:35:34.224728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224673    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy"
Nov 03 03:35:34.224728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224686    5515 cpu_manager.go:414] "containerMap: removing container" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy"
Nov 03 03:35:34.224728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224692    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy"
Nov 03 03:35:34.224836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224769    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy"
Nov 03 03:35:34.224836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.224777    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb11c27-bf88-4249-8603-f41209cc645b" containerName="istio-proxy"
Nov 03 03:35:34.229658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.229637    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.285988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.285957    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"]
Nov 03 03:35:34.333680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.333648    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a6b9ff\""
Nov 03 03:35:34.333875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.333704    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a6b9ff-dockercfg-8hc7k\""
Nov 03 03:35:34.365554 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.365528    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjgv\" (UniqueName: \"kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.365786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.365764    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.466801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.466766    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjgv\" (UniqueName: \"kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.466972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.466908    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.467573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.467536    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.529040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.528975    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjgv\" (UniqueName: \"kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.543754 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.543730    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:34.924085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:34.924058    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"]
Nov 03 03:35:34.924498 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:35:34.924470    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8b56d5_82d1_4f74_b849_90b8a7b2a53d.slice/crio-4ee3997c6a01ddab22b671299660300b79894e195fe5534a0e28ad38c57b8810 WatchSource:0}: Error finding container 4ee3997c6a01ddab22b671299660300b79894e195fe5534a0e28ad38c57b8810: Status 404 returned error can't find the container with id 4ee3997c6a01ddab22b671299660300b79894e195fe5534a0e28ad38c57b8810
Nov 03 03:35:35.405122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:35.405088    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:35.885594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:35.885542    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2" event={"ID":"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d","Type":"ContainerStarted","Data":"082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb"}
Nov 03 03:35:35.885801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:35.885620    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2" event={"ID":"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d","Type":"ContainerStarted","Data":"4ee3997c6a01ddab22b671299660300b79894e195fe5534a0e28ad38c57b8810"}
Nov 03 03:35:35.885801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:35.885713    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:36.005871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:36.005825    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2" podStartSLOduration=2.005811519 podStartE2EDuration="2.005811519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:36.003645177 +0000 UTC m=+2027.234537597" watchObservedRunningTime="2026-11-03 03:35:36.005811519 +0000 UTC m=+2027.236703941"
Nov 03 03:35:37.479418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:37.479383    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:35:39.052811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:39.052778    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:39.551275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:39.551238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:40.699575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.695419    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"]
Nov 03 03:35:40.715578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.714449    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:40.737029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.736995    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"]
Nov 03 03:35:40.790608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.785662    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95ab58-dockercfg-sd7kg\""
Nov 03 03:35:40.790608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.785759    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95ab58\""
Nov 03 03:35:40.834351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.834308    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:40.834531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.834481    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfvt\" (UniqueName: \"kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:40.935349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.935317    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfvt\" (UniqueName: \"kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:40.935589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.935392    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:40.936124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:40.936103    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:41.012359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.012269    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfvt\" (UniqueName: \"kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt\") pod \"payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:41.031161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.031127    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:41.460664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.460632    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"]
Nov 03 03:35:41.507781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.507746    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"]
Nov 03 03:35:41.508133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.508111    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.526248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.526217    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"]
Nov 03 03:35:41.650257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.650230    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f44lp\" (UniqueName: \"kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp\") pod \"payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.650359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.650337    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.751435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.751352    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f44lp\" (UniqueName: \"kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp\") pod \"payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.751435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.751406    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.752197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.752176    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.815052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.815002    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f44lp\" (UniqueName: \"kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp\") pod \"payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.833627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.833601    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:41.950221 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.950181    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2" event={"ID":"35572b6a-d024-4527-95f6-e43afd5d7349","Type":"ContainerStarted","Data":"0e5b582ccd2c8e189d1be8d9bb425bcbc5ac5b56a3532e396c8051fa7c982393"}
Nov 03 03:35:41.950419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.950392    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2" event={"ID":"35572b6a-d024-4527-95f6-e43afd5d7349","Type":"ContainerStarted","Data":"e189e6275b5be8f09adf42e7ced6d21969a60c7c0a68624760fffbc9fc43d6be"}
Nov 03 03:35:41.951349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:41.951332    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:42.119850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:42.119789    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2" podStartSLOduration=2.119768144 podStartE2EDuration="2.119768144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:42.102977017 +0000 UTC m=+2033.333869437" watchObservedRunningTime="2026-11-03 03:35:42.119768144 +0000 UTC m=+2033.350660564"
Nov 03 03:35:42.227005 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:42.226968    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:42.275153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:42.269870    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"]
Nov 03 03:35:42.959237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:42.959207    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6" event={"ID":"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c","Type":"ContainerStarted","Data":"19ac6c83259fa304014a1c39b8808d496623bbc43e682cb4258cbd245d7b4d67"}
Nov 03 03:35:42.959674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:42.959253    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6" event={"ID":"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c","Type":"ContainerStarted","Data":"92baeefca6c0257e8c20f9b70f59c7b4d6271123326dcac0e84dd33d145be2bf"}
Nov 03 03:35:42.959674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:42.959300    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:43.090876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:43.090810    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6" podStartSLOduration=2.090790493 podStartE2EDuration="2.090790493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:35:43.088839384 +0000 UTC m=+2034.319731805" watchObservedRunningTime="2026-11-03 03:35:43.090790493 +0000 UTC m=+2034.321682912"
Nov 03 03:35:45.162173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:45.162138    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:46.896565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:46.896504    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:35:47.049185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:47.049128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:48.028445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.028413    5515 generic.go:356] "Generic (PLEG): container finished" podID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerID="fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17" exitCode=0
Nov 03 03:35:48.028953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.028489    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" event={"ID":"5c240ad0-f62f-4516-8824-c710d0e960c9","Type":"ContainerDied","Data":"fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"}
Nov 03 03:35:48.028953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.028944    5515 scope.go:122] "RemoveContainer" containerID="fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"
Nov 03 03:35:48.030429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.030400    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:35:48.030568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.030473    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:35:48.030568 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.030503    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:35:48.147172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:48.147129    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:35:49.036138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.036088    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" event={"ID":"5c240ad0-f62f-4516-8824-c710d0e960c9","Type":"ContainerStarted","Data":"da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91"}
Nov 03 03:35:49.037569 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.037531    5515 generic.go:356] "Generic (PLEG): container finished" podID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerID="efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2" exitCode=0
Nov 03 03:35:49.037705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.037588    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" event={"ID":"f46f4684-5004-4dfe-aaaf-bb878fbf0e57","Type":"ContainerDied","Data":"efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"}
Nov 03 03:35:49.037969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.037953    5515 scope.go:122] "RemoveContainer" containerID="efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"
Nov 03 03:35:49.039311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.039284    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:35:49.039388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.039350    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:35:49.039388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.039380    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:35:49.287787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.287318    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:35:49.288666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.288254    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 03:35:49.288666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.288301    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 03:35:49.305934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:49.305912    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:35:50.050746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:50.050711    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" event={"ID":"f46f4684-5004-4dfe-aaaf-bb878fbf0e57","Type":"ContainerStarted","Data":"2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5"}
Nov 03 03:35:50.283487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:50.283005    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 03:35:50.283487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:50.283072    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 03:35:50.395287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:50.395253    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:35:50.403735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:50.403695    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 03 03:35:50.404042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:50.404004    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 03 03:35:51.032405 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:35:51.032369    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:35:51.287634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:51.286609    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 03:35:51.287634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:51.286666    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 03:35:51.386644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:51.386084    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 03 03:35:51.386644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:51.386144    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 03 03:35:51.789576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:51.788582    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:51.942522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:51.942485    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:52.092446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:52.092355    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:52.285725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:52.285692    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 03:35:52.285996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:52.285967    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 03:35:52.386990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:52.386955    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 03 03:35:52.387612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:52.387581    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 03 03:35:53.295574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:53.295079    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 03:35:53.295574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:53.295155    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 03:35:53.389989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:53.389952    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 03 03:35:53.390621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:53.390590    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 03 03:35:54.007737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:54.007701    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:35:54.322496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:54.322409    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:35:54.322986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:54.322963    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:35:54.324028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:54.324006    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:35:54.386921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:54.386887    5515 patch_prober.go:29] interesting pod/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 03 03:35:54.387086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:54.386948    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 03 03:35:55.007594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:55.007533    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:35:55.391063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:55.391030    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:35:55.391335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:55.391315    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:35:55.392187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:55.392166    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:35:56.978057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:56.978007    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:35:59.334679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:35:59.334635    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:03.621118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:03.621053    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:06.362675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:06.362639    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:06.844490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:06.844413    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:36:08.484573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:08.483745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:12.638791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:12.638727    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:13.399464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:13.399345    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:13.547080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:13.547021    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:13.704242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:13.704143    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:14.967756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:14.967722    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:17.046462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:17.046415    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:17.603675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:17.603270    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"]
Nov 03 03:36:17.715634 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:17.715533    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"]
Nov 03 03:36:17.971567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:17.971526    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:18.139870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.139834    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:18.306707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.306625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:18.387173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.387143    5515 generic.go:356] "Generic (PLEG): container finished" podID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerID="c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e" exitCode=0
Nov 03 03:36:18.387415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.387223    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" event={"ID":"9dcad0ae-8c13-431c-a3f1-ac2733e2b488","Type":"ContainerDied","Data":"c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"}
Nov 03 03:36:18.387785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.387763    5515 scope.go:122] "RemoveContainer" containerID="c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"
Nov 03 03:36:18.389579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.389511    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:36:18.389673 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.389623    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:36:18.389718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.389669    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:36:18.839680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:18.839651    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:19.397837 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:19.397800    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" event={"ID":"9dcad0ae-8c13-431c-a3f1-ac2733e2b488","Type":"ContainerStarted","Data":"fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce"}
Nov 03 03:36:19.507237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:19.507200    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:36:19.507658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:19.507631    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 03:36:19.507830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:19.507806    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 03:36:20.507385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:20.507347    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 03:36:20.507832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:20.507402    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 03:36:21.009441 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:21.009409    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:36:21.506973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:21.506942    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 03:36:21.507170 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:21.506992    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 03:36:21.597532 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:21.597499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:21.809893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:21.808174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:22.205713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.205665    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"]
Nov 03 03:36:22.209132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.209111    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.253811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.253769    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"]
Nov 03 03:36:22.301125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.301070    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6e985-dockercfg-xmssg\""
Nov 03 03:36:22.301125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.301084    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6e985\""
Nov 03 03:36:22.334754 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.334722    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.334936 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.334778    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbnq\" (UniqueName: \"kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq\") pod \"payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.436143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.436057    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbnq\" (UniqueName: \"kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq\") pod \"payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.436318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.436251    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.436938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.436919    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.498640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.498564    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbnq\" (UniqueName: \"kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq\") pod \"payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.507435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.507403    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 03:36:22.507597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.507478    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 03:36:22.524896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.524867    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:22.603835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.603790    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l" podUID="2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" containerName="payload-processing" containerID="cri-o://f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4" gracePeriod=25
Nov 03 03:36:22.716065 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.715983    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz" podUID="0ed2befc-5860-43e2-b5b9-954c9b452da8" containerName="payload-pre-processing" containerID="cri-o://4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452" gracePeriod=25
Nov 03 03:36:22.904008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.903982    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:36:22.937452 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:22.937411    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"]
Nov 03 03:36:22.938814 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:36:22.938785    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a150354_3572_40e1_b2e9_f5108f845895.slice/crio-eefd97eac4e42ef878c6b366a5a71f366148507b1ed1c17e6c55be8a6fe8e0e3 WatchSource:0}: Error finding container eefd97eac4e42ef878c6b366a5a71f366148507b1ed1c17e6c55be8a6fe8e0e3: Status 404 returned error can't find the container with id eefd97eac4e42ef878c6b366a5a71f366148507b1ed1c17e6c55be8a6fe8e0e3
Nov 03 03:36:23.020587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.020531    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:36:23.048580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.048494    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume\") pod \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") "
Nov 03 03:36:23.048739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.048718    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p\" (UniqueName: \"kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p\") pod \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\" (UID: \"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16\") "
Nov 03 03:36:23.048888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.048867    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume" pod "2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" (UID: "2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:36:23.049372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.049339    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:23.050998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.050970    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p" pod "2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" (UID: "2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16"). InnerVolumeSpecName "kube-api-access-hwp8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:36:23.150325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.150287    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w\" (UniqueName: \"kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w\") pod \"0ed2befc-5860-43e2-b5b9-954c9b452da8\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") "
Nov 03 03:36:23.150502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.150363    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume\") pod \"0ed2befc-5860-43e2-b5b9-954c9b452da8\" (UID: \"0ed2befc-5860-43e2-b5b9-954c9b452da8\") "
Nov 03 03:36:23.150791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.150766    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume" pod "0ed2befc-5860-43e2-b5b9-954c9b452da8" (UID: "0ed2befc-5860-43e2-b5b9-954c9b452da8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:36:23.150935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.150915    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed2befc-5860-43e2-b5b9-954c9b452da8-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:23.150990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.150942    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwp8p\" (UniqueName: \"kubernetes.io/projected/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16-kube-api-access-hwp8p\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:23.152948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.152926    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w" pod "0ed2befc-5860-43e2-b5b9-954c9b452da8" (UID: "0ed2befc-5860-43e2-b5b9-954c9b452da8"). InnerVolumeSpecName "kube-api-access-ln57w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:36:23.256046 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.255997    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln57w\" (UniqueName: \"kubernetes.io/projected/0ed2befc-5860-43e2-b5b9-954c9b452da8-kube-api-access-ln57w\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:23.423534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.423495    5515 generic.go:356] "Generic (PLEG): container finished" podID="2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" containerID="f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4" exitCode=0
Nov 03 03:36:23.423534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.423561    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"
Nov 03 03:36:23.423846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.423585    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l" event={"ID":"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16","Type":"ContainerDied","Data":"f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4"}
Nov 03 03:36:23.423846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.423649    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l" event={"ID":"2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16","Type":"ContainerDied","Data":"36f1bb9b252e5808a8b7df344d8fb96093a24855d8598de0684e8555ad2ec69d"}
Nov 03 03:36:23.423846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.423679    5515 scope.go:122] "RemoveContainer" containerID="f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4"
Nov 03 03:36:23.425255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.425233    5515 generic.go:356] "Generic (PLEG): container finished" podID="0ed2befc-5860-43e2-b5b9-954c9b452da8" containerID="4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452" exitCode=0
Nov 03 03:36:23.425376 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.425359    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"
Nov 03 03:36:23.425425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.425363    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz" event={"ID":"0ed2befc-5860-43e2-b5b9-954c9b452da8","Type":"ContainerDied","Data":"4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452"}
Nov 03 03:36:23.425462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.425439    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz" event={"ID":"0ed2befc-5860-43e2-b5b9-954c9b452da8","Type":"ContainerDied","Data":"c04d45453e8a56a0b82c5f732a48db4985d1f4b7ff925a28c4845de68dad8124"}
Nov 03 03:36:23.427290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.427264    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh" event={"ID":"2a150354-3572-40e1-b2e9-f5108f845895","Type":"ContainerStarted","Data":"08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81"}
Nov 03 03:36:23.427406 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.427315    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh" event={"ID":"2a150354-3572-40e1-b2e9-f5108f845895","Type":"ContainerStarted","Data":"eefd97eac4e42ef878c6b366a5a71f366148507b1ed1c17e6c55be8a6fe8e0e3"}
Nov 03 03:36:23.427406 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.427347    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:23.434925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.434905    5515 scope.go:122] "RemoveContainer" containerID="f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4"
Nov 03 03:36:23.435183 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:23.435162    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4\": container with ID starting with f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4 not found: ID does not exist" containerID="f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4"
Nov 03 03:36:23.435250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.435188    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4"} err="failed to get container status \"f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4\": rpc error: code = NotFound desc = could not find container \"f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4\": container with ID starting with f83f53263394dc5a25b991e47abeb18d7d691536d24373f769cce9be40ac03b4 not found: ID does not exist"
Nov 03 03:36:23.435250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.435202    5515 scope.go:122] "RemoveContainer" containerID="4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452"
Nov 03 03:36:23.445727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.445708    5515 scope.go:122] "RemoveContainer" containerID="4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452"
Nov 03 03:36:23.446005 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:23.445989    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452\": container with ID starting with 4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452 not found: ID does not exist" containerID="4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452"
Nov 03 03:36:23.446080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.446011    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452"} err="failed to get container status \"4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452\": rpc error: code = NotFound desc = could not find container \"4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452\": container with ID starting with 4189a47f7fc86f7df2cb5675bafc0306c6a4cc08e047c8fa386d4c3666ca3452 not found: ID does not exist"
Nov 03 03:36:23.491467 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:23.491433    5515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed2befc_5860_43e2_b5b9_954c9b452da8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5a582b_1d3a_4a76_86dc_5d27b9eb6f16.slice/crio-36f1bb9b252e5808a8b7df344d8fb96093a24855d8598de0684e8555ad2ec69d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5a582b_1d3a_4a76_86dc_5d27b9eb6f16.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 03:36:23.507362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.507337    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 03:36:23.507510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.507383    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 03:36:23.601987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.601959    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"]
Nov 03 03:36:23.633418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.633386    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-66c966989f-sgklz"]
Nov 03 03:36:23.716776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.716650    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"]
Nov 03 03:36:23.720594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.720568    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78984f6c89-5tg9l"]
Nov 03 03:36:23.843926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:23.843868    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh" podStartSLOduration=1.843853441 podStartE2EDuration="1.843853441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:23.842961119 +0000 UTC m=+2075.073853538" watchObservedRunningTime="2026-11-03 03:36:23.843853441 +0000 UTC m=+2075.074745870"
Nov 03 03:36:24.228693 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:24.228660    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:24.507351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:24.507265    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 03:36:24.507351 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:24.507319    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 03:36:24.530851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:24.530803    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:36:24.896207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:24.896176    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed2befc-5860-43e2-b5b9-954c9b452da8" path="/var/lib/kubelet/pods/0ed2befc-5860-43e2-b5b9-954c9b452da8/volumes"
Nov 03 03:36:24.896662 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:24.896648    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" path="/var/lib/kubelet/pods/2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16/volumes"
Nov 03 03:36:25.511374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:25.511341    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:36:25.512515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:25.512488    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:36:28.957056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:28.957023    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:29.102508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:29.102484    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:29.516055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:29.516025    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:29.677694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:29.677656    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:29.901933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:29.901903    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:33.161507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:33.161477    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:33.476450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:33.476374    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"]
Nov 03 03:36:33.593708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:33.593671    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"]
Nov 03 03:36:33.832726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:33.832655    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:33.979398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:33.979363    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:34.127922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:34.127888    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:34.444496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:34.444426    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:36:34.827624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:34.817399    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:35.055050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:35.055018    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:35.201577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:35.201535    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:38.477654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.477617    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx" podUID="0e12cdc0-9324-46c1-b447-c10cc19401cc" containerName="payload-processing" containerID="cri-o://02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47" gracePeriod=25
Nov 03 03:36:38.594750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.594716    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7" podUID="464c2bed-831e-4193-bc77-27b480a6a28e" containerName="payload-pre-processing" containerID="cri-o://45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4" gracePeriod=25
Nov 03 03:36:38.752792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.752769    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:36:38.785150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.785113    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr\" (UniqueName: \"kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr\") pod \"0e12cdc0-9324-46c1-b447-c10cc19401cc\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") "
Nov 03 03:36:38.785304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.785282    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume\") pod \"0e12cdc0-9324-46c1-b447-c10cc19401cc\" (UID: \"0e12cdc0-9324-46c1-b447-c10cc19401cc\") "
Nov 03 03:36:38.785763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.785729    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume" pod "0e12cdc0-9324-46c1-b447-c10cc19401cc" (UID: "0e12cdc0-9324-46c1-b447-c10cc19401cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:36:38.786074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.786019    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e12cdc0-9324-46c1-b447-c10cc19401cc-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:38.787731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.787688    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr" pod "0e12cdc0-9324-46c1-b447-c10cc19401cc" (UID: "0e12cdc0-9324-46c1-b447-c10cc19401cc"). InnerVolumeSpecName "kube-api-access-x5wmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:36:38.846565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.846527    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:36:38.887054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.887025    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume\") pod \"464c2bed-831e-4193-bc77-27b480a6a28e\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") "
Nov 03 03:36:38.887220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.887155    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj\" (UniqueName: \"kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj\") pod \"464c2bed-831e-4193-bc77-27b480a6a28e\" (UID: \"464c2bed-831e-4193-bc77-27b480a6a28e\") "
Nov 03 03:36:38.887429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.887405    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume" pod "464c2bed-831e-4193-bc77-27b480a6a28e" (UID: "464c2bed-831e-4193-bc77-27b480a6a28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:36:38.887822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.887803    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wmr\" (UniqueName: \"kubernetes.io/projected/0e12cdc0-9324-46c1-b447-c10cc19401cc-kube-api-access-x5wmr\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:38.887873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.887828    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/464c2bed-831e-4193-bc77-27b480a6a28e-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:38.889397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.889373    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj" pod "464c2bed-831e-4193-bc77-27b480a6a28e" (UID: "464c2bed-831e-4193-bc77-27b480a6a28e"). InnerVolumeSpecName "kube-api-access-nflcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:36:38.989156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:38.989099    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflcj\" (UniqueName: \"kubernetes.io/projected/464c2bed-831e-4193-bc77-27b480a6a28e-kube-api-access-nflcj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:36:39.096136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.096096    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:39.265231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.265151    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:39.422146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.422097    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:39.561150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.561062    5515 generic.go:356] "Generic (PLEG): container finished" podID="464c2bed-831e-4193-bc77-27b480a6a28e" containerID="45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4" exitCode=0
Nov 03 03:36:39.561150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.561139    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"
Nov 03 03:36:39.561721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.561155    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7" event={"ID":"464c2bed-831e-4193-bc77-27b480a6a28e","Type":"ContainerDied","Data":"45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4"}
Nov 03 03:36:39.561721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.561232    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7" event={"ID":"464c2bed-831e-4193-bc77-27b480a6a28e","Type":"ContainerDied","Data":"fbaea89f24e10cdf78538166d5b8210279f94808280c2c69fdfd747f68a91a86"}
Nov 03 03:36:39.561721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.561267    5515 scope.go:122] "RemoveContainer" containerID="45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4"
Nov 03 03:36:39.563232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.563121    5515 generic.go:356] "Generic (PLEG): container finished" podID="0e12cdc0-9324-46c1-b447-c10cc19401cc" containerID="02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47" exitCode=0
Nov 03 03:36:39.563232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.563183    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx" event={"ID":"0e12cdc0-9324-46c1-b447-c10cc19401cc","Type":"ContainerDied","Data":"02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47"}
Nov 03 03:36:39.563232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.563223    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx" event={"ID":"0e12cdc0-9324-46c1-b447-c10cc19401cc","Type":"ContainerDied","Data":"68142758e5bbed251c92a9710980970e042cd2ff6a22379d286f1db53797b874"}
Nov 03 03:36:39.563415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.563308    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"
Nov 03 03:36:39.575067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.575044    5515 scope.go:122] "RemoveContainer" containerID="45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4"
Nov 03 03:36:39.575426 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:39.575398    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4\": container with ID starting with 45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4 not found: ID does not exist" containerID="45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4"
Nov 03 03:36:39.575521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.575436    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4"} err="failed to get container status \"45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4\": rpc error: code = NotFound desc = could not find container \"45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4\": container with ID starting with 45e5238992bc0e0ff74395b488c6b41a4bf054af590b18219540672a37a94de4 not found: ID does not exist"
Nov 03 03:36:39.575521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.575462    5515 scope.go:122] "RemoveContainer" containerID="02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47"
Nov 03 03:36:39.585482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.585463    5515 scope.go:122] "RemoveContainer" containerID="02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47"
Nov 03 03:36:39.588670 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:39.588645    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47\": container with ID starting with 02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47 not found: ID does not exist" containerID="02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47"
Nov 03 03:36:39.588758 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.588677    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47"} err="failed to get container status \"02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47\": rpc error: code = NotFound desc = could not find container \"02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47\": container with ID starting with 02b8d7129b1fb6b995e0668808b781d95cc06b5d9e71ea0724b43270c7970b47 not found: ID does not exist"
Nov 03 03:36:39.731116 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.731082    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"]
Nov 03 03:36:39.781675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.781650    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-160d39-697dd65f59-8vkqx"]
Nov 03 03:36:39.844886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.844809    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"]
Nov 03 03:36:39.849320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:39.849298    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-160d39-57847879-jh2c7"]
Nov 03 03:36:40.546322 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:40.546290    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:40.894059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:40.894028    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e12cdc0-9324-46c1-b447-c10cc19401cc" path="/var/lib/kubelet/pods/0e12cdc0-9324-46c1-b447-c10cc19401cc/volumes"
Nov 03 03:36:40.894461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:40.894447    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464c2bed-831e-4193-bc77-27b480a6a28e" path="/var/lib/kubelet/pods/464c2bed-831e-4193-bc77-27b480a6a28e/volumes"
Nov 03 03:36:42.321408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.321374    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:42.502627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.502594    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:42.937336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.937299    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"]
Nov 03 03:36:42.938478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.938452    5515 cpu_manager.go:414] "containerMap: removing container" podUID="2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" containerName="payload-processing"
Nov 03 03:36:42.938652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.938636    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" containerName="payload-processing"
Nov 03 03:36:42.938774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.938760    5515 cpu_manager.go:414] "containerMap: removing container" podUID="464c2bed-831e-4193-bc77-27b480a6a28e" containerName="payload-pre-processing"
Nov 03 03:36:42.938860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.938848    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464c2bed-831e-4193-bc77-27b480a6a28e" containerName="payload-pre-processing"
Nov 03 03:36:42.938952 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.938943    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0e12cdc0-9324-46c1-b447-c10cc19401cc" containerName="payload-processing"
Nov 03 03:36:42.939029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939019    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e12cdc0-9324-46c1-b447-c10cc19401cc" containerName="payload-processing"
Nov 03 03:36:42.939112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939102    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0ed2befc-5860-43e2-b5b9-954c9b452da8" containerName="payload-pre-processing"
Nov 03 03:36:42.939209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939197    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed2befc-5860-43e2-b5b9-954c9b452da8" containerName="payload-pre-processing"
Nov 03 03:36:42.939509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939494    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5a582b-1d3a-4a76-86dc-5d27b9eb6f16" containerName="payload-processing"
Nov 03 03:36:42.939642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939630    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="464c2bed-831e-4193-bc77-27b480a6a28e" containerName="payload-pre-processing"
Nov 03 03:36:42.939726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939716    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e12cdc0-9324-46c1-b447-c10cc19401cc" containerName="payload-processing"
Nov 03 03:36:42.939794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.939786    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed2befc-5860-43e2-b5b9-954c9b452da8" containerName="payload-pre-processing"
Nov 03 03:36:42.943643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.943622    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:42.978978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:42.978943    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"]
Nov 03 03:36:43.020630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.020603    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8200d4\""
Nov 03 03:36:43.020805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.020641    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8200d4-dockercfg-p7kdv\""
Nov 03 03:36:43.051321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.051290    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.051627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.051602    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvz5\" (UniqueName: \"kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5\") pod \"payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.055246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.055224    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"]
Nov 03 03:36:43.058240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.058224    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.117037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.117009    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"]
Nov 03 03:36:43.153312 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.153274    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.153466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.153388    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvz5\" (UniqueName: \"kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5\") pod \"payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.153466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.153432    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphwg\" (UniqueName: \"kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.153623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.153486    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.155599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.154644    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.215313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.215247    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvz5\" (UniqueName: \"kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5\") pod \"payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.256758 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.256727    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphwg\" (UniqueName: \"kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.256907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.256778    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.257473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.257455    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.290211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.290185    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:43.319207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.319167    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphwg\" (UniqueName: \"kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.371195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.371168    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:43.668702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.668675    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"]
Nov 03 03:36:43.670051 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:36:43.670027    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffe0558_0da7_4d20_aa12_75ac4cfeaaeb.slice/crio-dd87e19c56b585c386d476179e7851c5264ad94210656fcf266160f6c478cef4 WatchSource:0}: Error finding container dd87e19c56b585c386d476179e7851c5264ad94210656fcf266160f6c478cef4: Status 404 returned error can't find the container with id dd87e19c56b585c386d476179e7851c5264ad94210656fcf266160f6c478cef4
Nov 03 03:36:43.768780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:43.768753    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"]
Nov 03 03:36:43.769210 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:36:43.769185    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27fa964_1336_4a0b_8656_81a845ccd64d.slice/crio-1267d5e8ee523126aa6c5d57af2f68a5009db5585fdbf2b438ac0fba2869fc32 WatchSource:0}: Error finding container 1267d5e8ee523126aa6c5d57af2f68a5009db5585fdbf2b438ac0fba2869fc32: Status 404 returned error can't find the container with id 1267d5e8ee523126aa6c5d57af2f68a5009db5585fdbf2b438ac0fba2869fc32
Nov 03 03:36:44.599974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.599891    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj" event={"ID":"f27fa964-1336-4a0b-8656-81a845ccd64d","Type":"ContainerStarted","Data":"a580b1496b0cc22824ca4d6aa2fb4f6f494a8a55a12bbe3972369e5254be1daa"}
Nov 03 03:36:44.599974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.599949    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj" event={"ID":"f27fa964-1336-4a0b-8656-81a845ccd64d","Type":"ContainerStarted","Data":"1267d5e8ee523126aa6c5d57af2f68a5009db5585fdbf2b438ac0fba2869fc32"}
Nov 03 03:36:44.600477 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.600103    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:44.601714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.601689    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" event={"ID":"affe0558-0da7-4d20-aa12-75ac4cfeaaeb","Type":"ContainerStarted","Data":"d7ff72e7fcb0778e2aba4cf46dd85aab9cc15d22277f6f781ce57ae00f8c916e"}
Nov 03 03:36:44.601828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.601730    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:44.601828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.601746    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" event={"ID":"affe0558-0da7-4d20-aa12-75ac4cfeaaeb","Type":"ContainerStarted","Data":"dd87e19c56b585c386d476179e7851c5264ad94210656fcf266160f6c478cef4"}
Nov 03 03:36:44.718327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.718277    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj" podStartSLOduration=1.71826348 podStartE2EDuration="1.71826348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:44.716708994 +0000 UTC m=+2095.947601412" watchObservedRunningTime="2026-11-03 03:36:44.71826348 +0000 UTC m=+2095.949155897"
Nov 03 03:36:44.848265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:44.835928    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" podStartSLOduration=2.835908035 podStartE2EDuration="2.835908035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:36:44.835214909 +0000 UTC m=+2096.066107328" watchObservedRunningTime="2026-11-03 03:36:44.835908035 +0000 UTC m=+2096.066800454"
Nov 03 03:36:47.679633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:47.679601    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:50.013534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:50.013503    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:36:50.015763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:50.015740    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:36:50.018896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:50.018877    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:36:50.021050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:50.021034    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:36:50.602029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:50.601928    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:36:50.984983 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:36:50.984948    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:36:52.885938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:52.885897    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:55.612316 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:55.612285    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:36:55.612791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:55.612363    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:36:58.026936 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:58.026907    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:36:59.693605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:36:59.693535    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:03.042030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:03.041992    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:08.584959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:08.584926    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"]
Nov 03 03:37:08.695122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:08.695060    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"]
Nov 03 03:37:08.857203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:08.857167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:09.019591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:09.019533    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:09.170827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:09.170729    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:10.134588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:10.134539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:13.585200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.585163    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" podUID="53e50ace-b076-4aea-bee9-1f2952628c29" containerName="payload-processing" containerID="cri-o://1ec8f6f2ae7ba357c17081a46e6cee2257986477e105cd23af8ea43d2c890f67" gracePeriod=25
Nov 03 03:37:13.604183 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:37:13.604140    5515 logging.go:55] [core] [Channel #3314 SubChannel #3315]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.126:50052", ServerName: "10.134.0.126:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.126:50052: connect: connection refused"
Nov 03 03:37:13.695839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.695801    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd" podUID="095f5fb8-5926-49c4-91da-d92ed249965c" containerName="payload-pre-processing" containerID="cri-o://f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319" gracePeriod=25
Nov 03 03:37:13.763780 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:37:13.763747    5515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095f5fb8_5926_49c4_91da_d92ed249965c.slice/crio-conmon-f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095f5fb8_5926_49c4_91da_d92ed249965c.slice/crio-f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 03:37:13.792855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.792822    5515 generic.go:356] "Generic (PLEG): container finished" podID="53e50ace-b076-4aea-bee9-1f2952628c29" containerID="1ec8f6f2ae7ba357c17081a46e6cee2257986477e105cd23af8ea43d2c890f67" exitCode=0
Nov 03 03:37:13.792986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.792899    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" event={"ID":"53e50ace-b076-4aea-bee9-1f2952628c29","Type":"ContainerDied","Data":"1ec8f6f2ae7ba357c17081a46e6cee2257986477e105cd23af8ea43d2c890f67"}
Nov 03 03:37:13.877043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.877019    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:37:13.963915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.963889    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume\") pod \"53e50ace-b076-4aea-bee9-1f2952628c29\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") "
Nov 03 03:37:13.964058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.964014    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt\" (UniqueName: \"kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt\") pod \"53e50ace-b076-4aea-bee9-1f2952628c29\" (UID: \"53e50ace-b076-4aea-bee9-1f2952628c29\") "
Nov 03 03:37:13.964260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.964235    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume" pod "53e50ace-b076-4aea-bee9-1f2952628c29" (UID: "53e50ace-b076-4aea-bee9-1f2952628c29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:37:13.964428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.964413    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e50ace-b076-4aea-bee9-1f2952628c29-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:37:13.964864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.964838    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:37:13.966173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:13.966153    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt" pod "53e50ace-b076-4aea-bee9-1f2952628c29" (UID: "53e50ace-b076-4aea-bee9-1f2952628c29"). InnerVolumeSpecName "kube-api-access-w87vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:37:14.065473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.065445    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume\") pod \"095f5fb8-5926-49c4-91da-d92ed249965c\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") "
Nov 03 03:37:14.065637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.065593    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85\" (UniqueName: \"kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85\") pod \"095f5fb8-5926-49c4-91da-d92ed249965c\" (UID: \"095f5fb8-5926-49c4-91da-d92ed249965c\") "
Nov 03 03:37:14.065859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.065835    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume" pod "095f5fb8-5926-49c4-91da-d92ed249965c" (UID: "095f5fb8-5926-49c4-91da-d92ed249965c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:37:14.065989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.065976    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w87vt\" (UniqueName: \"kubernetes.io/projected/53e50ace-b076-4aea-bee9-1f2952628c29-kube-api-access-w87vt\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:37:14.066037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.065991    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095f5fb8-5926-49c4-91da-d92ed249965c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:37:14.067844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.067821    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85" pod "095f5fb8-5926-49c4-91da-d92ed249965c" (UID: "095f5fb8-5926-49c4-91da-d92ed249965c"). InnerVolumeSpecName "kube-api-access-m9h85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:37:14.167515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.167470    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9h85\" (UniqueName: \"kubernetes.io/projected/095f5fb8-5926-49c4-91da-d92ed249965c-kube-api-access-m9h85\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:37:14.603978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.603879    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" podUID="53e50ace-b076-4aea-bee9-1f2952628c29" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.126:50052\" within 1s: context deadline exceeded"
Nov 03 03:37:14.801566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.801513    5515 generic.go:356] "Generic (PLEG): container finished" podID="095f5fb8-5926-49c4-91da-d92ed249965c" containerID="f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319" exitCode=0
Nov 03 03:37:14.801742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.801583    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd" event={"ID":"095f5fb8-5926-49c4-91da-d92ed249965c","Type":"ContainerDied","Data":"f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319"}
Nov 03 03:37:14.801742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.801596    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"
Nov 03 03:37:14.801742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.801630    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd" event={"ID":"095f5fb8-5926-49c4-91da-d92ed249965c","Type":"ContainerDied","Data":"00bc19341dddcd246d39f6b2960cc45a1d8aac8338f4345231c31987865c0239"}
Nov 03 03:37:14.801742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.801651    5515 scope.go:122] "RemoveContainer" containerID="f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319"
Nov 03 03:37:14.803514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.803488    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"
Nov 03 03:37:14.803670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.803646    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb" event={"ID":"53e50ace-b076-4aea-bee9-1f2952628c29","Type":"ContainerDied","Data":"435af77f6379f2a6d3ca8610ab57ce58f81f3bd4851f509bda4a6ad5ca499eb3"}
Nov 03 03:37:14.813530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.813482    5515 scope.go:122] "RemoveContainer" containerID="f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319"
Nov 03 03:37:14.813784 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:37:14.813766    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319\": container with ID starting with f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319 not found: ID does not exist" containerID="f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319"
Nov 03 03:37:14.813844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.813792    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319"} err="failed to get container status \"f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319\": rpc error: code = NotFound desc = could not find container \"f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319\": container with ID starting with f18f9fbbaea21f6eaaedd65e93c140b5947328d28c78424612c1b16b1647f319 not found: ID does not exist"
Nov 03 03:37:14.813844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.813807    5515 scope.go:122] "RemoveContainer" containerID="1ec8f6f2ae7ba357c17081a46e6cee2257986477e105cd23af8ea43d2c890f67"
Nov 03 03:37:14.969481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.969452    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"]
Nov 03 03:37:14.973956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:14.973935    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6ef2c-54fc74fd5f-sljkb"]
Nov 03 03:37:15.083813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:15.083790    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"]
Nov 03 03:37:15.090132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:15.090106    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6ef2c-7c749658c9-rg2qd"]
Nov 03 03:37:15.566224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:15.566183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:16.894519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:16.894488    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095f5fb8-5926-49c4-91da-d92ed249965c" path="/var/lib/kubelet/pods/095f5fb8-5926-49c4-91da-d92ed249965c/volumes"
Nov 03 03:37:16.894928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:16.894907    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e50ace-b076-4aea-bee9-1f2952628c29" path="/var/lib/kubelet/pods/53e50ace-b076-4aea-bee9-1f2952628c29/volumes"
Nov 03 03:37:19.451905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:19.451875    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:19.948855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:19.948828    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:20.984734 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:37:20.984701    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:37:21.050935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.050907    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"]
Nov 03 03:37:21.051542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.051527    5515 cpu_manager.go:414] "containerMap: removing container" podUID="095f5fb8-5926-49c4-91da-d92ed249965c" containerName="payload-pre-processing"
Nov 03 03:37:21.051542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.051558    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095f5fb8-5926-49c4-91da-d92ed249965c" containerName="payload-pre-processing"
Nov 03 03:37:21.051653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.051593    5515 cpu_manager.go:414] "containerMap: removing container" podUID="53e50ace-b076-4aea-bee9-1f2952628c29" containerName="payload-processing"
Nov 03 03:37:21.051653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.051600    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e50ace-b076-4aea-bee9-1f2952628c29" containerName="payload-processing"
Nov 03 03:37:21.051740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.051730    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="095f5fb8-5926-49c4-91da-d92ed249965c" containerName="payload-pre-processing"
Nov 03 03:37:21.051774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.051744    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e50ace-b076-4aea-bee9-1f2952628c29" containerName="payload-processing"
Nov 03 03:37:21.056819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.056802    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.127078 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.127044    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"]
Nov 03 03:37:21.160439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.160399    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f1c469\""
Nov 03 03:37:21.160439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.160436    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f1c469-dockercfg-54j7m\""
Nov 03 03:37:21.171912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.171889    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8vt\" (UniqueName: \"kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.172014 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.171933    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.273196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.273115    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8vt\" (UniqueName: \"kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.273196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.273158    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.273935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.273910    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.335541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.335504    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8vt\" (UniqueName: \"kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt\") pod \"payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.370517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.370490    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.748809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.748774    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"]
Nov 03 03:37:21.750060 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:37:21.750039    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec21374d_9c83_4a16_b0e4_aa003fec104b.slice/crio-74ba814f102c76e8eef32f6af2edc562207f09f12e434b54c93bf5000bf7d39d WatchSource:0}: Error finding container 74ba814f102c76e8eef32f6af2edc562207f09f12e434b54c93bf5000bf7d39d: Status 404 returned error can't find the container with id 74ba814f102c76e8eef32f6af2edc562207f09f12e434b54c93bf5000bf7d39d
Nov 03 03:37:21.773070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.773046    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"]
Nov 03 03:37:21.778326 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.778308    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:21.859521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.859481    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5" event={"ID":"ec21374d-9c83-4a16-b0e4-aa003fec104b","Type":"ContainerStarted","Data":"6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e"}
Nov 03 03:37:21.859731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.859572    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5" event={"ID":"ec21374d-9c83-4a16-b0e4-aa003fec104b","Type":"ContainerStarted","Data":"74ba814f102c76e8eef32f6af2edc562207f09f12e434b54c93bf5000bf7d39d"}
Nov 03 03:37:21.859731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.859618    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:21.860272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.860250    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"]
Nov 03 03:37:21.880515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.880482    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:21.880700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.880531    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9h5f\" (UniqueName: \"kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f\") pod \"payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:21.982399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.982368    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:21.982580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.982414    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9h5f\" (UniqueName: \"kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f\") pod \"payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:21.983168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:21.983149    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:22.012884 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.012796    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5" podStartSLOduration=1.012784591 podStartE2EDuration="1.012784591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:22.011931441 +0000 UTC m=+2133.242823859" watchObservedRunningTime="2026-11-03 03:37:22.012784591 +0000 UTC m=+2133.243677009"
Nov 03 03:37:22.044670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.044648    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9h5f\" (UniqueName: \"kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f\") pod \"payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:22.100242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.100217    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:22.520761 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.520733    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"]
Nov 03 03:37:22.521425 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:37:22.521392    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7af91ca_f528_4c97_891b_89570bb6c1e8.slice/crio-f79a87243af91f45e8d782c152e348e16021518a71bfc92c685d9b73bc9ff958 WatchSource:0}: Error finding container f79a87243af91f45e8d782c152e348e16021518a71bfc92c685d9b73bc9ff958: Status 404 returned error can't find the container with id f79a87243af91f45e8d782c152e348e16021518a71bfc92c685d9b73bc9ff958
Nov 03 03:37:22.835262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.835189    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:22.870624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.868571    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j" event={"ID":"b7af91ca-f528-4c97-891b-89570bb6c1e8","Type":"ContainerStarted","Data":"b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606"}
Nov 03 03:37:22.870624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.868631    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j" event={"ID":"b7af91ca-f528-4c97-891b-89570bb6c1e8","Type":"ContainerStarted","Data":"f79a87243af91f45e8d782c152e348e16021518a71bfc92c685d9b73bc9ff958"}
Nov 03 03:37:22.870624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.868791    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:22.991040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:22.990983    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j" podStartSLOduration=1.990964286 podStartE2EDuration="1.990964286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:22.990748534 +0000 UTC m=+2134.221640953" watchObservedRunningTime="2026-11-03 03:37:22.990964286 +0000 UTC m=+2134.221856705"
Nov 03 03:37:27.555096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:27.555064    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:27.718907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:27.718874    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:27.865789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:27.865757    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:32.639032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:32.638992    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:32.870991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:32.870957    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:37:33.880928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:33.880902    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:37:36.947752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:36.947690    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:37.787953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:37.787921    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:39.203598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:39.203544    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:39.348976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:39.348945    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:41.392248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:41.392212    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:41.890827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:41.890799    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:43.171639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:43.171527    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:44.723427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:44.723395    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:47.144353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:47.144321    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:37:50.085915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:50.085883    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:50.984378 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:37:50.984342    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:37:53.141968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:53.141880    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:54.037101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:54.037068    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:54.229563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:54.229524    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"]
Nov 03 03:37:54.485358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:54.485298    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:54.629863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:54.629824    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:54.777420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:54.777339    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:55.070420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:55.070334    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:55.362810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:55.362778    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:55.523533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:55.523500    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:55.670639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:55.667469    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:59.230156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.230098    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2" podUID="7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" containerName="payload-pre-processing" containerID="cri-o://082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb" gracePeriod=25
Nov 03 03:37:59.491433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.491373    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:37:59.653948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.653909    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume\") pod \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") "
Nov 03 03:37:59.654173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.654156    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv\" (UniqueName: \"kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv\") pod \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\" (UID: \"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d\") "
Nov 03 03:37:59.654266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.654249    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume" pod "7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" (UID: "7d8b56d5-82d1-4f74-b849-90b8a7b2a53d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:37:59.654679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.654660    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:37:59.656579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.656530    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv" pod "7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" (UID: "7d8b56d5-82d1-4f74-b849-90b8a7b2a53d"). InnerVolumeSpecName "kube-api-access-zsjgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:37:59.700710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.700688    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:37:59.756165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.756097    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsjgv\" (UniqueName: \"kubernetes.io/projected/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d-kube-api-access-zsjgv\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:37:59.846280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:37:59.846238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:00.042895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.042810    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:00.130152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.130124    5515 generic.go:356] "Generic (PLEG): container finished" podID="7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" containerID="082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb" exitCode=0
Nov 03 03:38:00.130308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.130172    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"
Nov 03 03:38:00.130308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.130204    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2" event={"ID":"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d","Type":"ContainerDied","Data":"082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb"}
Nov 03 03:38:00.130308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.130258    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2" event={"ID":"7d8b56d5-82d1-4f74-b849-90b8a7b2a53d","Type":"ContainerDied","Data":"4ee3997c6a01ddab22b671299660300b79894e195fe5534a0e28ad38c57b8810"}
Nov 03 03:38:00.130308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.130285    5515 scope.go:122] "RemoveContainer" containerID="082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb"
Nov 03 03:38:00.140543 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.140527    5515 scope.go:122] "RemoveContainer" containerID="082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb"
Nov 03 03:38:00.140851 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:00.140833    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb\": container with ID starting with 082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb not found: ID does not exist" containerID="082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb"
Nov 03 03:38:00.140902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.140860    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb"} err="failed to get container status \"082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb\": rpc error: code = NotFound desc = could not find container \"082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb\": container with ID starting with 082ec8d2d8367e27a36ceb6f996233ba6d2bb580fa0d9c61b10083f886e559cb not found: ID does not exist"
Nov 03 03:38:00.300648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.300542    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"]
Nov 03 03:38:00.350842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.350809    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7b99f49595-fj9v2"]
Nov 03 03:38:00.801449 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.801403    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:00.895801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:00.895770    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" path="/var/lib/kubelet/pods/7d8b56d5-82d1-4f74-b849-90b8a7b2a53d/volumes"
Nov 03 03:38:03.560977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:03.560937    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:03.753710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:03.753657    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:08.015724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:08.015688    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:12.833057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:12.833022    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:16.808364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:16.808329    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:16.956962 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:16.956929    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:17.242165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:17.242131    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:38:17.914683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:17.914650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:20.984652 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:20.984617    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:38:21.550233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:21.550203    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:22.046178 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:22.046146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:23.969261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:23.969229    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"]
Nov 03 03:38:23.969738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:23.969431    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" podUID="195f97df-261f-4a58-8855-4b1fa447d7eb" containerName="maas-api" containerID="cri-o://39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23" gracePeriod=30
Nov 03 03:38:24.076110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.076070    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:24.293857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.293824    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:38:24.301179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.301144    5515 generic.go:356] "Generic (PLEG): container finished" podID="195f97df-261f-4a58-8855-4b1fa447d7eb" containerID="39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23" exitCode=0
Nov 03 03:38:24.301303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.301177    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"
Nov 03 03:38:24.301303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.301212    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" event={"ID":"195f97df-261f-4a58-8855-4b1fa447d7eb","Type":"ContainerDied","Data":"39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23"}
Nov 03 03:38:24.301303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.301267    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94" event={"ID":"195f97df-261f-4a58-8855-4b1fa447d7eb","Type":"ContainerDied","Data":"06a3fd479c347a936a3356ed915c889fd3c925b3bb4b05d41e530fa7e9f200e6"}
Nov 03 03:38:24.301303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.301285    5515 scope.go:122] "RemoveContainer" containerID="39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23"
Nov 03 03:38:24.311320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.311302    5515 scope.go:122] "RemoveContainer" containerID="39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23"
Nov 03 03:38:24.311624 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:24.311605    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23\": container with ID starting with 39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23 not found: ID does not exist" containerID="39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23"
Nov 03 03:38:24.311707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.311631    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23"} err="failed to get container status \"39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23\": rpc error: code = NotFound desc = could not find container \"39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23\": container with ID starting with 39835a1d529e27a20d0c8c946e02c61249498153fdf3dbb3f5a5a1037c1cef23 not found: ID does not exist"
Nov 03 03:38:24.429917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.429880    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"]
Nov 03 03:38:24.430159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.430135    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc" podUID="6d5c93be-ba31-4cce-bbf1-11603e2eece4" containerName="payload-processing" containerID="cri-o://b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623" gracePeriod=30
Nov 03 03:38:24.448265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.448239    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs\") pod \"195f97df-261f-4a58-8855-4b1fa447d7eb\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") "
Nov 03 03:38:24.448389 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.448342    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5\" (UniqueName: \"kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5\") pod \"195f97df-261f-4a58-8855-4b1fa447d7eb\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") "
Nov 03 03:38:24.448455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.448418    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca\") pod \"195f97df-261f-4a58-8855-4b1fa447d7eb\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") "
Nov 03 03:38:24.448721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.448606    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\") pod \"195f97df-261f-4a58-8855-4b1fa447d7eb\" (UID: \"195f97df-261f-4a58-8855-4b1fa447d7eb\") "
Nov 03 03:38:24.449217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.449194    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca" pod "195f97df-261f-4a58-8855-4b1fa447d7eb" (UID: "195f97df-261f-4a58-8855-4b1fa447d7eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:38:24.451489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.451463    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls" pod "195f97df-261f-4a58-8855-4b1fa447d7eb" (UID: "195f97df-261f-4a58-8855-4b1fa447d7eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:38:24.451669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.451648    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs" pod "195f97df-261f-4a58-8855-4b1fa447d7eb" (UID: "195f97df-261f-4a58-8855-4b1fa447d7eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:38:24.452224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.452196    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5" pod "195f97df-261f-4a58-8855-4b1fa447d7eb" (UID: "195f97df-261f-4a58-8855-4b1fa447d7eb"). InnerVolumeSpecName "kube-api-access-6kvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:24.484241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.484219    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"]
Nov 03 03:38:24.484445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.484422    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874" podUID="a5013180-60c8-4188-9b82-3d00880d9d0f" containerName="payload-pre-processing" containerID="cri-o://2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0" gracePeriod=30
Nov 03 03:38:24.549923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.549894    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.549923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.549920    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvz5\" (UniqueName: \"kubernetes.io/projected/195f97df-261f-4a58-8855-4b1fa447d7eb-kube-api-access-6kvz5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.549923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.549931    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195f97df-261f-4a58-8855-4b1fa447d7eb-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.550195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.549940    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195f97df-261f-4a58-8855-4b1fa447d7eb-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.594790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.594756    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:24.750469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.750439    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:38:24.751612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.751589    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume\") pod \"a5013180-60c8-4188-9b82-3d00880d9d0f\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") "
Nov 03 03:38:24.751855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.751834    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj\" (UniqueName: \"kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj\") pod \"a5013180-60c8-4188-9b82-3d00880d9d0f\" (UID: \"a5013180-60c8-4188-9b82-3d00880d9d0f\") "
Nov 03 03:38:24.751946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.751864    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume" pod "a5013180-60c8-4188-9b82-3d00880d9d0f" (UID: "a5013180-60c8-4188-9b82-3d00880d9d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:38:24.752363 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.752340    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5013180-60c8-4188-9b82-3d00880d9d0f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.755195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.755168    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj" pod "a5013180-60c8-4188-9b82-3d00880d9d0f" (UID: "a5013180-60c8-4188-9b82-3d00880d9d0f"). InnerVolumeSpecName "kube-api-access-jxfmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:24.782418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.782401    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:38:24.789295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.789273    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:24.816166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.816138    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"]
Nov 03 03:38:24.819492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.819470    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a5156-7c6bd5656f-jmt94"]
Nov 03 03:38:24.853195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.853173    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume\") pod \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") "
Nov 03 03:38:24.853310 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.853245    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724\" (UniqueName: \"kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724\") pod \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\" (UID: \"6d5c93be-ba31-4cce-bbf1-11603e2eece4\") "
Nov 03 03:38:24.853602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.853575    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume" pod "6d5c93be-ba31-4cce-bbf1-11603e2eece4" (UID: "6d5c93be-ba31-4cce-bbf1-11603e2eece4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:38:24.853690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.853647    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxfmj\" (UniqueName: \"kubernetes.io/projected/a5013180-60c8-4188-9b82-3d00880d9d0f-kube-api-access-jxfmj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.855536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.855513    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724" pod "6d5c93be-ba31-4cce-bbf1-11603e2eece4" (UID: "6d5c93be-ba31-4cce-bbf1-11603e2eece4"). InnerVolumeSpecName "kube-api-access-6c724". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:24.893891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.893866    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195f97df-261f-4a58-8855-4b1fa447d7eb" path="/var/lib/kubelet/pods/195f97df-261f-4a58-8855-4b1fa447d7eb/volumes"
Nov 03 03:38:24.960676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.960604    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c724\" (UniqueName: \"kubernetes.io/projected/6d5c93be-ba31-4cce-bbf1-11603e2eece4-kube-api-access-6c724\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:24.960676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:24.960644    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d5c93be-ba31-4cce-bbf1-11603e2eece4-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:25.308202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.308167    5515 generic.go:356] "Generic (PLEG): container finished" podID="a5013180-60c8-4188-9b82-3d00880d9d0f" containerID="2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0" exitCode=0
Nov 03 03:38:25.308668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.308219    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"
Nov 03 03:38:25.308668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.308253    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874" event={"ID":"a5013180-60c8-4188-9b82-3d00880d9d0f","Type":"ContainerDied","Data":"2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0"}
Nov 03 03:38:25.308668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.308308    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874" event={"ID":"a5013180-60c8-4188-9b82-3d00880d9d0f","Type":"ContainerDied","Data":"b2e903303ecd37501f49a8181c9f765dde6c4768ab31601829f21167ab6444f7"}
Nov 03 03:38:25.308668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.308334    5515 scope.go:122] "RemoveContainer" containerID="2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0"
Nov 03 03:38:25.310094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.310064    5515 generic.go:356] "Generic (PLEG): container finished" podID="6d5c93be-ba31-4cce-bbf1-11603e2eece4" containerID="b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623" exitCode=0
Nov 03 03:38:25.310225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.310103    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"
Nov 03 03:38:25.310225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.310114    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc" event={"ID":"6d5c93be-ba31-4cce-bbf1-11603e2eece4","Type":"ContainerDied","Data":"b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623"}
Nov 03 03:38:25.310225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.310168    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc" event={"ID":"6d5c93be-ba31-4cce-bbf1-11603e2eece4","Type":"ContainerDied","Data":"1ae01a8da7507f150838bdac36ef364bf3b9f3cd427768091ede209ebf8e9bbf"}
Nov 03 03:38:25.318203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.318185    5515 scope.go:122] "RemoveContainer" containerID="2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0"
Nov 03 03:38:25.318523 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:25.318500    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0\": container with ID starting with 2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0 not found: ID does not exist" containerID="2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0"
Nov 03 03:38:25.318656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.318529    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0"} err="failed to get container status \"2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0\": rpc error: code = NotFound desc = could not find container \"2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0\": container with ID starting with 2ee4ab8c34e83bb19b549d3f1fc2760e45584b5fc6b95074811bb5140de77af0 not found: ID does not exist"
Nov 03 03:38:25.318656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.318568    5515 scope.go:122] "RemoveContainer" containerID="b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623"
Nov 03 03:38:25.327687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.327670    5515 scope.go:122] "RemoveContainer" containerID="b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623"
Nov 03 03:38:25.327906 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:25.327892    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623\": container with ID starting with b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623 not found: ID does not exist" containerID="b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623"
Nov 03 03:38:25.327977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.327913    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623"} err="failed to get container status \"b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623\": rpc error: code = NotFound desc = could not find container \"b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623\": container with ID starting with b554c8f691547859b22176e7d0784aeb8f2fc9f2bb2714772f03b4a131718623 not found: ID does not exist"
Nov 03 03:38:25.476160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.476120    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"]
Nov 03 03:38:25.480207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.480178    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a5156-6876848db6-95kgc"]
Nov 03 03:38:25.607230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.607203    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"]
Nov 03 03:38:25.610992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.610970    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a5156-5bbf887bcf-p2874"]
Nov 03 03:38:25.931506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:25.931378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:26.894413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:26.894377    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5c93be-ba31-4cce-bbf1-11603e2eece4" path="/var/lib/kubelet/pods/6d5c93be-ba31-4cce-bbf1-11603e2eece4/volumes"
Nov 03 03:38:26.894819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:26.894808    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5013180-60c8-4188-9b82-3d00880d9d0f" path="/var/lib/kubelet/pods/a5013180-60c8-4188-9b82-3d00880d9d0f/volumes"
Nov 03 03:38:30.929750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:30.929720    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:35.213303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:35.213264    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:35.359817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:35.359769    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:36.809340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:36.809302    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:37.314696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:37.314664    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"]
Nov 03 03:38:37.314912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:37.314888    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy" containerID="cri-o://6db15ebd10f13a30e8377dba12c8994cc490f6d4e30dbd581cdc446a3099ea10" gracePeriod=30
Nov 03 03:38:37.321238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:37.321212    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:37.750453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:37.750418    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:38.115905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:38.115872    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:38.513774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:38.513694    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:42.453542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.453510    5515 generic.go:356] "Generic (PLEG): container finished" podID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerID="6db15ebd10f13a30e8377dba12c8994cc490f6d4e30dbd581cdc446a3099ea10" exitCode=0
Nov 03 03:38:42.453964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.453579    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" event={"ID":"5127b2e0-c1fc-445c-a158-eea5dfab716e","Type":"ContainerDied","Data":"6db15ebd10f13a30e8377dba12c8994cc490f6d4e30dbd581cdc446a3099ea10"}
Nov 03 03:38:42.453964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.453650    5515 scope.go:122] "RemoveContainer" containerID="5f9b8d89faea9a865bec652633b858cbfc1b3bfbe6af02a12dc8b8d60d5ca6ac"
Nov 03 03:38:42.580401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.580334    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:38:42.677066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.677041    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:42.699850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.699827    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.699963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.699875    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.699963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.699939    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.699963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.699956    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.700100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.699998    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.700100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700034    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.700208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700116    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.700208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700163    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.700315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700249    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs\") pod \"5127b2e0-c1fc-445c-a158-eea5dfab716e\" (UID: \"5127b2e0-c1fc-445c-a158-eea5dfab716e\") "
Nov 03 03:38:42.700315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700250    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:38:42.700430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700326    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:38:42.700430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700337    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:38:42.700697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700534    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:38:42.700697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700596    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:38:42.700821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700756    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.700821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700775    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.700821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700788    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.700821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700801    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5127b2e0-c1fc-445c-a158-eea5dfab716e-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.700821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.700814    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.702511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.702482    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:38:42.702637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.702530    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "kube-api-access-sxtvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:42.702809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.702782    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:38:42.702809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.702796    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token" pod "5127b2e0-c1fc-445c-a158-eea5dfab716e" (UID: "5127b2e0-c1fc-445c-a158-eea5dfab716e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:42.802224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.802190    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.802224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.802221    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.802224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.802232    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5127b2e0-c1fc-445c-a158-eea5dfab716e-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:42.802449 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:42.802241    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtvz\" (UniqueName: \"kubernetes.io/projected/5127b2e0-c1fc-445c-a158-eea5dfab716e-kube-api-access-sxtvz\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:43.461921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:43.461889    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"
Nov 03 03:38:43.462353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:43.461914    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c" event={"ID":"5127b2e0-c1fc-445c-a158-eea5dfab716e","Type":"ContainerDied","Data":"6b87f91f41e5f55120af22cbbab6975d99d8c68e572162891a9093b5403fe373"}
Nov 03 03:38:43.462353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:43.461979    5515 scope.go:122] "RemoveContainer" containerID="6db15ebd10f13a30e8377dba12c8994cc490f6d4e30dbd581cdc446a3099ea10"
Nov 03 03:38:43.629830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:43.629792    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"]
Nov 03 03:38:43.680620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:43.680590    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4a5156-openshift-default-bdd7cdff9-w5p6c"]
Nov 03 03:38:44.578266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:44.578194    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:44.894533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:44.894497    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" path="/var/lib/kubelet/pods/5127b2e0-c1fc-445c-a158-eea5dfab716e/volumes"
Nov 03 03:38:45.086119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:45.086086    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:47.919007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:47.918969    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:50.985022 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:50.984975    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:38:51.800866 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:51.800834    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"]
Nov 03 03:38:51.985283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:51.985249    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:52.176214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:52.176180    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:52.320248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:52.320209    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:53.243443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:53.243410    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:38:56.801237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:56.801201    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh" podUID="2a150354-3572-40e1-b2e9-f5108f845895" containerName="payload-processing" containerID="cri-o://08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81" gracePeriod=25
Nov 03 03:38:57.071479 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.071450    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:38:57.175876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.175847    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume\") pod \"2a150354-3572-40e1-b2e9-f5108f845895\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") "
Nov 03 03:38:57.176050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.176030    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq\" (UniqueName: \"kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq\") pod \"2a150354-3572-40e1-b2e9-f5108f845895\" (UID: \"2a150354-3572-40e1-b2e9-f5108f845895\") "
Nov 03 03:38:57.176314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.176277    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume" pod "2a150354-3572-40e1-b2e9-f5108f845895" (UID: "2a150354-3572-40e1-b2e9-f5108f845895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:38:57.176498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.176481    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a150354-3572-40e1-b2e9-f5108f845895-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:57.178495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.178466    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq" pod "2a150354-3572-40e1-b2e9-f5108f845895" (UID: "2a150354-3572-40e1-b2e9-f5108f845895"). InnerVolumeSpecName "kube-api-access-qkbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:57.277690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.277662    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbnq\" (UniqueName: \"kubernetes.io/projected/2a150354-3572-40e1-b2e9-f5108f845895-kube-api-access-qkbnq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:38:57.547516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.547482    5515 generic.go:356] "Generic (PLEG): container finished" podID="2a150354-3572-40e1-b2e9-f5108f845895" containerID="08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81" exitCode=0
Nov 03 03:38:57.547748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.547533    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"
Nov 03 03:38:57.547748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.547583    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh" event={"ID":"2a150354-3572-40e1-b2e9-f5108f845895","Type":"ContainerDied","Data":"08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81"}
Nov 03 03:38:57.547748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.547638    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh" event={"ID":"2a150354-3572-40e1-b2e9-f5108f845895","Type":"ContainerDied","Data":"eefd97eac4e42ef878c6b366a5a71f366148507b1ed1c17e6c55be8a6fe8e0e3"}
Nov 03 03:38:57.547748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.547651    5515 scope.go:122] "RemoveContainer" containerID="08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81"
Nov 03 03:38:57.557739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.557716    5515 scope.go:122] "RemoveContainer" containerID="08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81"
Nov 03 03:38:57.557992 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:38:57.557973    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81\": container with ID starting with 08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81 not found: ID does not exist" containerID="08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81"
Nov 03 03:38:57.558039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.558001    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81"} err="failed to get container status \"08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81\": rpc error: code = NotFound desc = could not find container \"08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81\": container with ID starting with 08399c5cff8ad92b30e7d6142afd7b440f91b8be70ba85b1791f03ce84ea8a81 not found: ID does not exist"
Nov 03 03:38:57.715269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.715236    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"]
Nov 03 03:38:57.766015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:57.765990    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-5479d58d99-ln7sh"]
Nov 03 03:38:58.894108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:38:58.894079    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a150354-3572-40e1-b2e9-f5108f845895" path="/var/lib/kubelet/pods/2a150354-3572-40e1-b2e9-f5108f845895/volumes"
Nov 03 03:39:00.010902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:00.010867    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:00.431939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:00.431891    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:03.619805 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:03.619772    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:04.124319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:04.124284    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:04.345920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:04.345882    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:39:05.225127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225094    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"]
Nov 03 03:39:05.225680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225665    5515 cpu_manager.go:414] "containerMap: removing container" podUID="7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" containerName="payload-pre-processing"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225681    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" containerName="payload-pre-processing"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225704    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6d5c93be-ba31-4cce-bbf1-11603e2eece4" containerName="payload-processing"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225709    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5c93be-ba31-4cce-bbf1-11603e2eece4" containerName="payload-processing"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225718    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a5013180-60c8-4188-9b82-3d00880d9d0f" containerName="payload-pre-processing"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225723    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5013180-60c8-4188-9b82-3d00880d9d0f" containerName="payload-pre-processing"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225732    5515 cpu_manager.go:414] "containerMap: removing container" podUID="195f97df-261f-4a58-8855-4b1fa447d7eb" containerName="maas-api"
Nov 03 03:39:05.225734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225736    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195f97df-261f-4a58-8855-4b1fa447d7eb" containerName="maas-api"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225743    5515 cpu_manager.go:414] "containerMap: removing container" podUID="2a150354-3572-40e1-b2e9-f5108f845895" containerName="payload-processing"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225748    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a150354-3572-40e1-b2e9-f5108f845895" containerName="payload-processing"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225762    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225766    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225774    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225779    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225868    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5c93be-ba31-4cce-bbf1-11603e2eece4" containerName="payload-processing"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225879    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5013180-60c8-4188-9b82-3d00880d9d0f" containerName="payload-pre-processing"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225885    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="195f97df-261f-4a58-8855-4b1fa447d7eb" containerName="maas-api"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225891    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a150354-3572-40e1-b2e9-f5108f845895" containerName="payload-processing"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225897    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225903    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5127b2e0-c1fc-445c-a158-eea5dfab716e" containerName="istio-proxy"
Nov 03 03:39:05.225954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.225913    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8b56d5-82d1-4f74-b849-90b8a7b2a53d" containerName="payload-pre-processing"
Nov 03 03:39:05.230421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.230405    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.281240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.281212    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.281371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.281359    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfgj\" (UniqueName: \"kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.287392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.287368    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"]
Nov 03 03:39:05.334606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.334575    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a6b9ff\""
Nov 03 03:39:05.334606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.334601    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a6b9ff-dockercfg-gnxx9\""
Nov 03 03:39:05.383064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.383034    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.383245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.383226    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfgj\" (UniqueName: \"kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.383839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.383823    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.445057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.445030    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfgj\" (UniqueName: \"kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj\") pod \"payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.544471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.544398    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:05.924798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:05.924761    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"]
Nov 03 03:39:05.925808 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:39:05.925780    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13f8abc_7373_455b_b065_5262e76c5949.slice/crio-5296e8b09029d77fc80bd659039facc30b7f13e075e4529049a61ca2b92f56ee WatchSource:0}: Error finding container 5296e8b09029d77fc80bd659039facc30b7f13e075e4529049a61ca2b92f56ee: Status 404 returned error can't find the container with id 5296e8b09029d77fc80bd659039facc30b7f13e075e4529049a61ca2b92f56ee
Nov 03 03:39:06.020179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.020148    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"]
Nov 03 03:39:06.024245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.024226    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.082891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.082864    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"]
Nov 03 03:39:06.091270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.091241    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.091380 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.091364    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbrw\" (UniqueName: \"kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw\") pod \"payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.192744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.192670    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.192744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.192720    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbrw\" (UniqueName: \"kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw\") pod \"payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.193395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.193369    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.255044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.255018    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbrw\" (UniqueName: \"kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw\") pod \"payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.337460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.337433    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:06.617875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.617837    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4" event={"ID":"d13f8abc-7373-455b-b065-5262e76c5949","Type":"ContainerStarted","Data":"b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0"}
Nov 03 03:39:06.618055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.617911    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:06.618055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.617934    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4" event={"ID":"d13f8abc-7373-455b-b065-5262e76c5949","Type":"ContainerStarted","Data":"5296e8b09029d77fc80bd659039facc30b7f13e075e4529049a61ca2b92f56ee"}
Nov 03 03:39:06.716085 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.716058    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"]
Nov 03 03:39:06.716921 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:39:06.716895    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc45778_8f2e_4ff7_b200_5233d868dbe9.slice/crio-6544abf30102e1cb119b1577422d1e3f8d3706b79c3c96b4280d32afa7274655 WatchSource:0}: Error finding container 6544abf30102e1cb119b1577422d1e3f8d3706b79c3c96b4280d32afa7274655: Status 404 returned error can't find the container with id 6544abf30102e1cb119b1577422d1e3f8d3706b79c3c96b4280d32afa7274655
Nov 03 03:39:06.741736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.741691    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4" podStartSLOduration=1.741679097 podStartE2EDuration="1.741679097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:39:06.739118148 +0000 UTC m=+2237.970010566" watchObservedRunningTime="2026-11-03 03:39:06.741679097 +0000 UTC m=+2237.972571560"
Nov 03 03:39:06.823704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:06.823641    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:07.626325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:07.626292    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj" event={"ID":"abc45778-8f2e-4ff7-b200-5233d868dbe9","Type":"ContainerStarted","Data":"5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995"}
Nov 03 03:39:07.626853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:07.626344    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj" event={"ID":"abc45778-8f2e-4ff7-b200-5233d868dbe9","Type":"ContainerStarted","Data":"6544abf30102e1cb119b1577422d1e3f8d3706b79c3c96b4280d32afa7274655"}
Nov 03 03:39:07.626853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:07.626576    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:07.747716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:07.747650    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj" podStartSLOduration=2.747636656 podStartE2EDuration="2.747636656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:39:07.744648645 +0000 UTC m=+2238.975541075" watchObservedRunningTime="2026-11-03 03:39:07.747636656 +0000 UTC m=+2238.978529111"
Nov 03 03:39:11.629670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:11.629637    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:16.186896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.186857    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"]
Nov 03 03:39:16.191158 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.191140    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.298832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.298803    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b9c10a-serving-cert\""
Nov 03 03:39:16.301559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.301525    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"]
Nov 03 03:39:16.312678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.312643    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.312788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.312758    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.312844 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.312832    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.312886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.312870    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvsv\" (UniqueName: \"kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.397104 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.397065    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:16.414620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.414593    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.414730 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.414645    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvsv\" (UniqueName: \"kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.414772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.414742    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.414843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.414829    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.414884 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:16.414744    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-serving-cert: secret "maas-api-e2e-ipp-b-b9c10a-serving-cert" not found
Nov 03 03:39:16.414957 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:16.414935    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls podName:6bddcb24-f783-4859-9b74-98bd5404b9a2 nodeName:}" failed. No retries permitted until 2026-11-03 03:39:16.914916602 +0000 UTC m=+2248.145808998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls") pod "maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" (UID: "6bddcb24-f783-4859-9b74-98bd5404b9a2") : secret "maas-api-e2e-ipp-b-b9c10a-serving-cert" not found
Nov 03 03:39:16.415441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.415419    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.417504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.417484    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.493175 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.493060    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvsv\" (UniqueName: \"kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.566567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.566503    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:16.722481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.722455    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:16.922335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.922309    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:16.924845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:16.924819    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:17.104425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:17.104390    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:17.481537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:17.481508    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"]
Nov 03 03:39:17.482733 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:39:17.482700    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bddcb24_f783_4859_9b74_98bd5404b9a2.slice/crio-dd86b13d3e2d47433d797166a0c70a1c84602f6ecc971b4622e81f4f2f3b5975 WatchSource:0}: Error finding container dd86b13d3e2d47433d797166a0c70a1c84602f6ecc971b4622e81f4f2f3b5975: Status 404 returned error can't find the container with id dd86b13d3e2d47433d797166a0c70a1c84602f6ecc971b4622e81f4f2f3b5975
Nov 03 03:39:17.484224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:17.484204    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:39:17.628688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:17.628659    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:39:17.689106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:17.689070    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" event={"ID":"6bddcb24-f783-4859-9b74-98bd5404b9a2","Type":"ContainerStarted","Data":"dd86b13d3e2d47433d797166a0c70a1c84602f6ecc971b4622e81f4f2f3b5975"}
Nov 03 03:39:18.242863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:18.242814    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:18.634701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:18.634673    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:39:18.696166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:18.696127    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" event={"ID":"6bddcb24-f783-4859-9b74-98bd5404b9a2","Type":"ContainerStarted","Data":"f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040"}
Nov 03 03:39:18.696313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:18.696195    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:18.878258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:18.878208    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" podStartSLOduration=1.9935333750000002 podStartE2EDuration="2.878194517s" totalImagesPullingTime="884.661142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:39:18.877108226 +0000 UTC m=+2250.108000645" watchObservedRunningTime="2026-11-03 03:39:18.878194517 +0000 UTC m=+2250.109086934"
Nov 03 03:39:20.024347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:20.024282    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:20.527039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:20.526997    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"]
Nov 03 03:39:20.637410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:20.637364    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"]
Nov 03 03:39:20.894111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:20.894080    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:21.014722 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:21.014685    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:39:21.046932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.046863    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:21.101722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.101677    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"]
Nov 03 03:39:21.102301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.102232    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" podUID="0f0a70da-3b42-4618-af6e-ead7001835af" containerName="maas-api" containerID="cri-o://84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09" gracePeriod=30
Nov 03 03:39:21.198593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.198489    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:21.415213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.415187    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:39:21.510313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.510236    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs\") pod \"0f0a70da-3b42-4618-af6e-ead7001835af\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") "
Nov 03 03:39:21.510447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.510380    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca\") pod \"0f0a70da-3b42-4618-af6e-ead7001835af\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") "
Nov 03 03:39:21.510447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.510421    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs\" (UniqueName: \"kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs\") pod \"0f0a70da-3b42-4618-af6e-ead7001835af\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") "
Nov 03 03:39:21.510543 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.510526    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\") pod \"0f0a70da-3b42-4618-af6e-ead7001835af\" (UID: \"0f0a70da-3b42-4618-af6e-ead7001835af\") "
Nov 03 03:39:21.510846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.510823    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca" pod "0f0a70da-3b42-4618-af6e-ead7001835af" (UID: "0f0a70da-3b42-4618-af6e-ead7001835af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:21.511089 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.511066    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f0a70da-3b42-4618-af6e-ead7001835af-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:21.512709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.512678    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs" pod "0f0a70da-3b42-4618-af6e-ead7001835af" (UID: "0f0a70da-3b42-4618-af6e-ead7001835af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:39:21.512709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.512695    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls" pod "0f0a70da-3b42-4618-af6e-ead7001835af" (UID: "0f0a70da-3b42-4618-af6e-ead7001835af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:39:21.512833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.512769    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs" pod "0f0a70da-3b42-4618-af6e-ead7001835af" (UID: "0f0a70da-3b42-4618-af6e-ead7001835af"). InnerVolumeSpecName "kube-api-access-gtsgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:21.543885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.543854    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"]
Nov 03 03:39:21.544397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.544121    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6" podUID="8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" containerName="payload-processing" containerID="cri-o://19ac6c83259fa304014a1c39b8808d496623bbc43e682cb4258cbd245d7b4d67" gracePeriod=30
Nov 03 03:39:21.594258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.594232    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"]
Nov 03 03:39:21.594436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.594417    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2" podUID="35572b6a-d024-4527-95f6-e43afd5d7349" containerName="payload-pre-processing" containerID="cri-o://0e5b582ccd2c8e189d1be8d9bb425bcbc5ac5b56a3532e396c8051fa7c982393" gracePeriod=30
Nov 03 03:39:21.612071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.612045    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:21.612071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.612071    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f0a70da-3b42-4618-af6e-ead7001835af-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:21.612204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.612086    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtsgs\" (UniqueName: \"kubernetes.io/projected/0f0a70da-3b42-4618-af6e-ead7001835af-kube-api-access-gtsgs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:21.700480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.700440    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:21.719691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.719609    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6" event={"ID":"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c","Type":"ContainerDied","Data":"19ac6c83259fa304014a1c39b8808d496623bbc43e682cb4258cbd245d7b4d67"}
Nov 03 03:39:21.720239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.720090    5515 generic.go:356] "Generic (PLEG): container finished" podID="8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" containerID="19ac6c83259fa304014a1c39b8808d496623bbc43e682cb4258cbd245d7b4d67" exitCode=0
Nov 03 03:39:21.722519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.722493    5515 generic.go:356] "Generic (PLEG): container finished" podID="35572b6a-d024-4527-95f6-e43afd5d7349" containerID="0e5b582ccd2c8e189d1be8d9bb425bcbc5ac5b56a3532e396c8051fa7c982393" exitCode=0
Nov 03 03:39:21.722675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.722656    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2" event={"ID":"35572b6a-d024-4527-95f6-e43afd5d7349","Type":"ContainerDied","Data":"0e5b582ccd2c8e189d1be8d9bb425bcbc5ac5b56a3532e396c8051fa7c982393"}
Nov 03 03:39:21.724310 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.724238    5515 generic.go:356] "Generic (PLEG): container finished" podID="0f0a70da-3b42-4618-af6e-ead7001835af" containerID="84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09" exitCode=0
Nov 03 03:39:21.724429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.724365    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" event={"ID":"0f0a70da-3b42-4618-af6e-ead7001835af","Type":"ContainerDied","Data":"84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09"}
Nov 03 03:39:21.724429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.724401    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc" event={"ID":"0f0a70da-3b42-4618-af6e-ead7001835af","Type":"ContainerDied","Data":"e93052e3547c9c1b139ca66dcf15cecc1e887499f5ec6fdcb922ea278b7ae821"}
Nov 03 03:39:21.724605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.724431    5515 scope.go:122] "RemoveContainer" containerID="84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09"
Nov 03 03:39:21.724681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.724649    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"
Nov 03 03:39:21.740882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.740819    5515 scope.go:122] "RemoveContainer" containerID="84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09"
Nov 03 03:39:21.741260 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:21.741207    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09\": container with ID starting with 84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09 not found: ID does not exist" containerID="84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09"
Nov 03 03:39:21.741260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.741237    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09"} err="failed to get container status \"84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09\": rpc error: code = NotFound desc = could not find container \"84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09\": container with ID starting with 84649dffc58d90a86d3ec2123a7657f140fd4fdf0f9a731c144bd6ab8e8ffb09 not found: ID does not exist"
Nov 03 03:39:21.848209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.848178    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:21.867214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.867189    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:39:21.885854 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.885831    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:39:21.905969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.905944    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"]
Nov 03 03:39:21.910562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:21.910529    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95ab58-76799fb8f5-nn5hc"]
Nov 03 03:39:22.017659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.017632    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp\" (UniqueName: \"kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp\") pod \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") "
Nov 03 03:39:22.017784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.017704    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt\" (UniqueName: \"kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt\") pod \"35572b6a-d024-4527-95f6-e43afd5d7349\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") "
Nov 03 03:39:22.017784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.017738    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume\") pod \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\" (UID: \"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c\") "
Nov 03 03:39:22.017859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.017808    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume\") pod \"35572b6a-d024-4527-95f6-e43afd5d7349\" (UID: \"35572b6a-d024-4527-95f6-e43afd5d7349\") "
Nov 03 03:39:22.018234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.018209    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume" pod "8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" (UID: "8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:22.018234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.018213    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume" pod "35572b6a-d024-4527-95f6-e43afd5d7349" (UID: "35572b6a-d024-4527-95f6-e43afd5d7349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:22.019998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.019977    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp" pod "8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" (UID: "8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c"). InnerVolumeSpecName "kube-api-access-f44lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:22.020074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.019997    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt" pod "35572b6a-d024-4527-95f6-e43afd5d7349" (UID: "35572b6a-d024-4527-95f6-e43afd5d7349"). InnerVolumeSpecName "kube-api-access-qmfvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:22.128625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.128440    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmfvt\" (UniqueName: \"kubernetes.io/projected/35572b6a-d024-4527-95f6-e43afd5d7349-kube-api-access-qmfvt\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:22.128625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.128482    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:22.128625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.128500    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35572b6a-d024-4527-95f6-e43afd5d7349-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:22.128625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.128515    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f44lp\" (UniqueName: \"kubernetes.io/projected/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c-kube-api-access-f44lp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:22.731795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.731765    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"
Nov 03 03:39:22.731795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.731786    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6" event={"ID":"8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c","Type":"ContainerDied","Data":"92baeefca6c0257e8c20f9b70f59c7b4d6271123326dcac0e84dd33d145be2bf"}
Nov 03 03:39:22.732052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.731848    5515 scope.go:122] "RemoveContainer" containerID="19ac6c83259fa304014a1c39b8808d496623bbc43e682cb4258cbd245d7b4d67"
Nov 03 03:39:22.733242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.733163    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"
Nov 03 03:39:22.733242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.733205    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2" event={"ID":"35572b6a-d024-4527-95f6-e43afd5d7349","Type":"ContainerDied","Data":"e189e6275b5be8f09adf42e7ced6d21969a60c7c0a68624760fffbc9fc43d6be"}
Nov 03 03:39:22.743041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.743006    5515 scope.go:122] "RemoveContainer" containerID="0e5b582ccd2c8e189d1be8d9bb425bcbc5ac5b56a3532e396c8051fa7c982393"
Nov 03 03:39:22.895155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.895124    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0a70da-3b42-4618-af6e-ead7001835af" path="/var/lib/kubelet/pods/0f0a70da-3b42-4618-af6e-ead7001835af/volumes"
Nov 03 03:39:22.899258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.899235    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"]
Nov 03 03:39:22.902861 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:22.902841    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95ab58-856c5b54c4-8fmf6"]
Nov 03 03:39:23.015654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:23.015588    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"]
Nov 03 03:39:23.023249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:23.023227    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95ab58-544b6f6775-sjnz2"]
Nov 03 03:39:23.045428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:23.045400    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:24.709769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:24.709740    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:39:24.899728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:24.899692    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35572b6a-d024-4527-95f6-e43afd5d7349" path="/var/lib/kubelet/pods/35572b6a-d024-4527-95f6-e43afd5d7349/volumes"
Nov 03 03:39:24.900349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:24.900326    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" path="/var/lib/kubelet/pods/8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c/volumes"
Nov 03 03:39:25.527793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.527755    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" podUID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" containerName="payload-processing" containerID="cri-o://d7ff72e7fcb0778e2aba4cf46dd85aab9cc15d22277f6f781ce57ae00f8c916e" gracePeriod=25
Nov 03 03:39:25.610645 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:39:25.610617    5515 logging.go:55] [core] [Channel #3637 SubChannel #3639]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: connect: connection refused"
Nov 03 03:39:25.638533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.638494    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj" podUID="f27fa964-1336-4a0b-8656-81a845ccd64d" containerName="payload-pre-processing" containerID="cri-o://a580b1496b0cc22824ca4d6aa2fb4f6f494a8a55a12bbe3972369e5254be1daa" gracePeriod=25
Nov 03 03:39:25.758206 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.758177    5515 generic.go:356] "Generic (PLEG): container finished" podID="f27fa964-1336-4a0b-8656-81a845ccd64d" containerID="a580b1496b0cc22824ca4d6aa2fb4f6f494a8a55a12bbe3972369e5254be1daa" exitCode=0
Nov 03 03:39:25.758668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.758262    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj" event={"ID":"f27fa964-1336-4a0b-8656-81a845ccd64d","Type":"ContainerDied","Data":"a580b1496b0cc22824ca4d6aa2fb4f6f494a8a55a12bbe3972369e5254be1daa"}
Nov 03 03:39:25.760369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.760345    5515 generic.go:356] "Generic (PLEG): container finished" podID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" containerID="d7ff72e7fcb0778e2aba4cf46dd85aab9cc15d22277f6f781ce57ae00f8c916e" exitCode=0
Nov 03 03:39:25.760480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.760429    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" event={"ID":"affe0558-0da7-4d20-aa12-75ac4cfeaaeb","Type":"ContainerDied","Data":"d7ff72e7fcb0778e2aba4cf46dd85aab9cc15d22277f6f781ce57ae00f8c916e"}
Nov 03 03:39:25.807516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.807492    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:39:25.878086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.878048    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5\" (UniqueName: \"kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5\") pod \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") "
Nov 03 03:39:25.878279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.878213    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume\") pod \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\" (UID: \"affe0558-0da7-4d20-aa12-75ac4cfeaaeb\") "
Nov 03 03:39:25.878601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.878576    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume" pod "affe0558-0da7-4d20-aa12-75ac4cfeaaeb" (UID: "affe0558-0da7-4d20-aa12-75ac4cfeaaeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:25.878687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.878677    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:25.880740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.880715    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5" pod "affe0558-0da7-4d20-aa12-75ac4cfeaaeb" (UID: "affe0558-0da7-4d20-aa12-75ac4cfeaaeb"). InnerVolumeSpecName "kube-api-access-9rvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:25.912166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.912145    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:39:25.979944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.979915    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume\") pod \"f27fa964-1336-4a0b-8656-81a845ccd64d\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") "
Nov 03 03:39:25.980119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.980002    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg\" (UniqueName: \"kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg\") pod \"f27fa964-1336-4a0b-8656-81a845ccd64d\" (UID: \"f27fa964-1336-4a0b-8656-81a845ccd64d\") "
Nov 03 03:39:25.980307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.980285    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume" pod "f27fa964-1336-4a0b-8656-81a845ccd64d" (UID: "f27fa964-1336-4a0b-8656-81a845ccd64d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:25.980601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.980541    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27fa964-1336-4a0b-8656-81a845ccd64d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:25.980694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.980607    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvz5\" (UniqueName: \"kubernetes.io/projected/affe0558-0da7-4d20-aa12-75ac4cfeaaeb-kube-api-access-9rvz5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:25.982246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:25.982224    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg" pod "f27fa964-1336-4a0b-8656-81a845ccd64d" (UID: "f27fa964-1336-4a0b-8656-81a845ccd64d"). InnerVolumeSpecName "kube-api-access-cphwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:26.081509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.081419    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cphwg\" (UniqueName: \"kubernetes.io/projected/f27fa964-1336-4a0b-8656-81a845ccd64d-kube-api-access-cphwg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:26.523946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.523917    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:26.610335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.610297    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" podUID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Nov 03 03:39:26.768040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.768003    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"
Nov 03 03:39:26.768482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.768020    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj" event={"ID":"f27fa964-1336-4a0b-8656-81a845ccd64d","Type":"ContainerDied","Data":"1267d5e8ee523126aa6c5d57af2f68a5009db5585fdbf2b438ac0fba2869fc32"}
Nov 03 03:39:26.768482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.768153    5515 scope.go:122] "RemoveContainer" containerID="a580b1496b0cc22824ca4d6aa2fb4f6f494a8a55a12bbe3972369e5254be1daa"
Nov 03 03:39:26.769642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.769443    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"
Nov 03 03:39:26.769642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.769485    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9" event={"ID":"affe0558-0da7-4d20-aa12-75ac4cfeaaeb","Type":"ContainerDied","Data":"dd87e19c56b585c386d476179e7851c5264ad94210656fcf266160f6c478cef4"}
Nov 03 03:39:26.779951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.779932    5515 scope.go:122] "RemoveContainer" containerID="d7ff72e7fcb0778e2aba4cf46dd85aab9cc15d22277f6f781ce57ae00f8c916e"
Nov 03 03:39:26.980462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:26.980425    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"]
Nov 03 03:39:27.031144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:27.031075    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-6d57b64679-xxmj9"]
Nov 03 03:39:27.120682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:27.120648    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"]
Nov 03 03:39:27.126327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:27.126302    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-78b987b899-pzgkj"]
Nov 03 03:39:27.475411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:27.475382    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:27.625179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:27.625150    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:28.165417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:28.165386    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:28.893762 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:28.893730    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" path="/var/lib/kubelet/pods/affe0558-0da7-4d20-aa12-75ac4cfeaaeb/volumes"
Nov 03 03:39:28.894127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:28.894113    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27fa964-1336-4a0b-8656-81a845ccd64d" path="/var/lib/kubelet/pods/f27fa964-1336-4a0b-8656-81a845ccd64d/volumes"
Nov 03 03:39:29.043630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:29.043594    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:29.528654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:29.528626    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:30.362817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:30.362788    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:33.715403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:33.715374    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:35.359915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:35.359880    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:39:37.114478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:37.114442    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:38.959007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:38.958971    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:39.106457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:39.106424    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:40.614612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:40.614575    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:45.031596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.031528    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"]
Nov 03 03:39:45.032058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.031820    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" podUID="42629e00-25a1-4c3a-ad4d-ececc0b29834" containerName="maas-api" containerID="cri-o://52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593" gracePeriod=30
Nov 03 03:39:45.133859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.133829    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:45.344260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.344235    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:39:45.444451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.444416    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp\" (UniqueName: \"kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp\") pod \"42629e00-25a1-4c3a-ad4d-ececc0b29834\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") "
Nov 03 03:39:45.444813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.444782    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs\") pod \"42629e00-25a1-4c3a-ad4d-ececc0b29834\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") "
Nov 03 03:39:45.445120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.445094    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca\") pod \"42629e00-25a1-4c3a-ad4d-ececc0b29834\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") "
Nov 03 03:39:45.445335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.445253    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\") pod \"42629e00-25a1-4c3a-ad4d-ececc0b29834\" (UID: \"42629e00-25a1-4c3a-ad4d-ececc0b29834\") "
Nov 03 03:39:45.447287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.447265    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca" pod "42629e00-25a1-4c3a-ad4d-ececc0b29834" (UID: "42629e00-25a1-4c3a-ad4d-ececc0b29834"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:45.451957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.451917    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp" pod "42629e00-25a1-4c3a-ad4d-ececc0b29834" (UID: "42629e00-25a1-4c3a-ad4d-ececc0b29834"). InnerVolumeSpecName "kube-api-access-sgsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:45.456042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.455996    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls" pod "42629e00-25a1-4c3a-ad4d-ececc0b29834" (UID: "42629e00-25a1-4c3a-ad4d-ececc0b29834"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:39:45.456293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.456264    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs" pod "42629e00-25a1-4c3a-ad4d-ececc0b29834" (UID: "42629e00-25a1-4c3a-ad4d-ececc0b29834"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:39:45.508922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.508883    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"]
Nov 03 03:39:45.509132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.509105    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j" podUID="b7af91ca-f528-4c97-891b-89570bb6c1e8" containerName="payload-processing" containerID="cri-o://b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606" gracePeriod=30
Nov 03 03:39:45.547651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.547624    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:45.547651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.547652    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42629e00-25a1-4c3a-ad4d-ececc0b29834-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:45.547792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.547662    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42629e00-25a1-4c3a-ad4d-ececc0b29834-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:45.547792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.547670    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsjp\" (UniqueName: \"kubernetes.io/projected/42629e00-25a1-4c3a-ad4d-ececc0b29834-kube-api-access-sgsjp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:45.563609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.563584    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"]
Nov 03 03:39:45.563783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.563765    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5" podUID="ec21374d-9c83-4a16-b0e4-aa003fec104b" containerName="payload-pre-processing" containerID="cri-o://6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e" gracePeriod=30
Nov 03 03:39:45.668249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.668148    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:45.791150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.791122    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:39:45.812919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.812895    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:45.850899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.850870    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume\") pod \"b7af91ca-f528-4c97-891b-89570bb6c1e8\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") "
Nov 03 03:39:45.851044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.850920    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f\" (UniqueName: \"kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f\") pod \"b7af91ca-f528-4c97-891b-89570bb6c1e8\" (UID: \"b7af91ca-f528-4c97-891b-89570bb6c1e8\") "
Nov 03 03:39:45.851286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.851257    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume" pod "b7af91ca-f528-4c97-891b-89570bb6c1e8" (UID: "b7af91ca-f528-4c97-891b-89570bb6c1e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:45.851544 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.851530    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7af91ca-f528-4c97-891b-89570bb6c1e8-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:45.853091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.853070    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f" pod "b7af91ca-f528-4c97-891b-89570bb6c1e8" (UID: "b7af91ca-f528-4c97-891b-89570bb6c1e8"). InnerVolumeSpecName "kube-api-access-p9h5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:45.857873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.857851    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:39:45.885852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.885830    5515 generic.go:356] "Generic (PLEG): container finished" podID="ec21374d-9c83-4a16-b0e4-aa003fec104b" containerID="6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e" exitCode=0
Nov 03 03:39:45.885957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.885870    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"
Nov 03 03:39:45.885957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.885918    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5" event={"ID":"ec21374d-9c83-4a16-b0e4-aa003fec104b","Type":"ContainerDied","Data":"6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e"}
Nov 03 03:39:45.886060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.885961    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5" event={"ID":"ec21374d-9c83-4a16-b0e4-aa003fec104b","Type":"ContainerDied","Data":"74ba814f102c76e8eef32f6af2edc562207f09f12e434b54c93bf5000bf7d39d"}
Nov 03 03:39:45.886060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.885984    5515 scope.go:122] "RemoveContainer" containerID="6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e"
Nov 03 03:39:45.887792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.887771    5515 generic.go:356] "Generic (PLEG): container finished" podID="42629e00-25a1-4c3a-ad4d-ececc0b29834" containerID="52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593" exitCode=0
Nov 03 03:39:45.887896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.887812    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"
Nov 03 03:39:45.887896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.887852    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" event={"ID":"42629e00-25a1-4c3a-ad4d-ececc0b29834","Type":"ContainerDied","Data":"52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593"}
Nov 03 03:39:45.887996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.887899    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9" event={"ID":"42629e00-25a1-4c3a-ad4d-ececc0b29834","Type":"ContainerDied","Data":"28ff49a4e496bf02f7caf74db9f1780bcf7a07c8f49f4156be4ac9f7d527ffff"}
Nov 03 03:39:45.891021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.891002    5515 generic.go:356] "Generic (PLEG): container finished" podID="b7af91ca-f528-4c97-891b-89570bb6c1e8" containerID="b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606" exitCode=0
Nov 03 03:39:45.891120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.891048    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j" event={"ID":"b7af91ca-f528-4c97-891b-89570bb6c1e8","Type":"ContainerDied","Data":"b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606"}
Nov 03 03:39:45.891120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.891059    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"
Nov 03 03:39:45.891120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.891085    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j" event={"ID":"b7af91ca-f528-4c97-891b-89570bb6c1e8","Type":"ContainerDied","Data":"f79a87243af91f45e8d782c152e348e16021518a71bfc92c685d9b73bc9ff958"}
Nov 03 03:39:45.905901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.905875    5515 scope.go:122] "RemoveContainer" containerID="6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e"
Nov 03 03:39:45.906343 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:45.906314    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e\": container with ID starting with 6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e not found: ID does not exist" containerID="6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e"
Nov 03 03:39:45.906427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.906355    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e"} err="failed to get container status \"6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e\": rpc error: code = NotFound desc = could not find container \"6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e\": container with ID starting with 6ba20047f44355be73f48e377e5199adf462980927b0a93403bdafdf12832e0e not found: ID does not exist"
Nov 03 03:39:45.906427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.906380    5515 scope.go:122] "RemoveContainer" containerID="52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593"
Nov 03 03:39:45.920287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.920243    5515 scope.go:122] "RemoveContainer" containerID="52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593"
Nov 03 03:39:45.920690 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:45.920663    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593\": container with ID starting with 52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593 not found: ID does not exist" containerID="52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593"
Nov 03 03:39:45.920767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.920702    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593"} err="failed to get container status \"52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593\": rpc error: code = NotFound desc = could not find container \"52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593\": container with ID starting with 52eb4683f5d30b80a6a4d7cbc83481d030edd19e9ece79d83349c0f5b5cc8593 not found: ID does not exist"
Nov 03 03:39:45.920767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.920726    5515 scope.go:122] "RemoveContainer" containerID="b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606"
Nov 03 03:39:45.930354 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.930335    5515 scope.go:122] "RemoveContainer" containerID="b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606"
Nov 03 03:39:45.930639 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:45.930614    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606\": container with ID starting with b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606 not found: ID does not exist" containerID="b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606"
Nov 03 03:39:45.930716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.930645    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606"} err="failed to get container status \"b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606\": rpc error: code = NotFound desc = could not find container \"b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606\": container with ID starting with b1b34f7055cad843d712c634515070383e64177cd1c89f04098d5f7d4b8ae606 not found: ID does not exist"
Nov 03 03:39:45.952851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.952721    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume\") pod \"ec21374d-9c83-4a16-b0e4-aa003fec104b\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") "
Nov 03 03:39:45.952851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.952813    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt\" (UniqueName: \"kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt\") pod \"ec21374d-9c83-4a16-b0e4-aa003fec104b\" (UID: \"ec21374d-9c83-4a16-b0e4-aa003fec104b\") "
Nov 03 03:39:45.953242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.953195    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume" pod "ec21374d-9c83-4a16-b0e4-aa003fec104b" (UID: "ec21374d-9c83-4a16-b0e4-aa003fec104b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:45.953416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.953344    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9h5f\" (UniqueName: \"kubernetes.io/projected/b7af91ca-f528-4c97-891b-89570bb6c1e8-kube-api-access-p9h5f\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:45.955221 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:45.955198    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt" pod "ec21374d-9c83-4a16-b0e4-aa003fec104b" (UID: "ec21374d-9c83-4a16-b0e4-aa003fec104b"). InnerVolumeSpecName "kube-api-access-hh8vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:46.058640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.058578    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec21374d-9c83-4a16-b0e4-aa003fec104b-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:46.058640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.058616    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh8vt\" (UniqueName: \"kubernetes.io/projected/ec21374d-9c83-4a16-b0e4-aa003fec104b-kube-api-access-hh8vt\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:46.251067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.251034    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"]
Nov 03 03:39:46.255207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.255181    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f1c469-6d7dfd98dc-67qx9"]
Nov 03 03:39:46.365704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.365670    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"]
Nov 03 03:39:46.370171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.370147    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f1c469-78c4cdf9d-jjg5j"]
Nov 03 03:39:46.481168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.481095    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"]
Nov 03 03:39:46.485385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.485359    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f1c469-b6dc7f847-99hw5"]
Nov 03 03:39:46.896127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.896098    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42629e00-25a1-4c3a-ad4d-ececc0b29834" path="/var/lib/kubelet/pods/42629e00-25a1-4c3a-ad4d-ececc0b29834/volumes"
Nov 03 03:39:46.896510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.896497    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7af91ca-f528-4c97-891b-89570bb6c1e8" path="/var/lib/kubelet/pods/b7af91ca-f528-4c97-891b-89570bb6c1e8/volumes"
Nov 03 03:39:46.896896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:46.896882    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec21374d-9c83-4a16-b0e4-aa003fec104b" path="/var/lib/kubelet/pods/ec21374d-9c83-4a16-b0e4-aa003fec104b/volumes"
Nov 03 03:39:47.024101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:47.024075    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:50.983848 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:50.983811    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:39:52.053873 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:52.053837    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:52.914624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:52.914593    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"]
Nov 03 03:39:52.914802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:52.914770    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy" containerID="cri-o://348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1" gracePeriod=30
Nov 03 03:39:55.657407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:55.657375    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:56.162049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:56.162017    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:56.428764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:56.428670    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:56.591575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:56.585028    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:57.256336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256304    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"]
Nov 03 03:39:57.256903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256884    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0f0a70da-3b42-4618-af6e-ead7001835af" containerName="maas-api"
Nov 03 03:39:57.256953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256905    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0a70da-3b42-4618-af6e-ead7001835af" containerName="maas-api"
Nov 03 03:39:57.256953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256922    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ec21374d-9c83-4a16-b0e4-aa003fec104b" containerName="payload-pre-processing"
Nov 03 03:39:57.256953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256927    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec21374d-9c83-4a16-b0e4-aa003fec104b" containerName="payload-pre-processing"
Nov 03 03:39:57.256953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256942    5515 cpu_manager.go:414] "containerMap: removing container" podUID="35572b6a-d024-4527-95f6-e43afd5d7349" containerName="payload-pre-processing"
Nov 03 03:39:57.256953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256947    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35572b6a-d024-4527-95f6-e43afd5d7349" containerName="payload-pre-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256956    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b7af91ca-f528-4c97-891b-89570bb6c1e8" containerName="payload-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256961    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7af91ca-f528-4c97-891b-89570bb6c1e8" containerName="payload-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256980    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f27fa964-1336-4a0b-8656-81a845ccd64d" containerName="payload-pre-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256985    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27fa964-1336-4a0b-8656-81a845ccd64d" containerName="payload-pre-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.256999    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" containerName="payload-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257004    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" containerName="payload-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257012    5515 cpu_manager.go:414] "containerMap: removing container" podUID="42629e00-25a1-4c3a-ad4d-ececc0b29834" containerName="maas-api"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257017    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42629e00-25a1-4c3a-ad4d-ececc0b29834" containerName="maas-api"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257022    5515 cpu_manager.go:414] "containerMap: removing container" podUID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" containerName="payload-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257026    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" containerName="payload-processing"
Nov 03 03:39:57.257102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257103    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7af91ca-f528-4c97-891b-89570bb6c1e8" containerName="payload-processing"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257112    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27fa964-1336-4a0b-8656-81a845ccd64d" containerName="payload-pre-processing"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257120    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d017307-cd8f-4ff7-abf8-e14b4ed0ab4c" containerName="payload-processing"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257127    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="42629e00-25a1-4c3a-ad4d-ececc0b29834" containerName="maas-api"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257134    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="affe0558-0da7-4d20-aa12-75ac4cfeaaeb" containerName="payload-processing"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257142    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0a70da-3b42-4618-af6e-ead7001835af" containerName="maas-api"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257148    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec21374d-9c83-4a16-b0e4-aa003fec104b" containerName="payload-pre-processing"
Nov 03 03:39:57.257420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.257155    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="35572b6a-d024-4527-95f6-e43afd5d7349" containerName="payload-pre-processing"
Nov 03 03:39:57.262431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.262411    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.310267 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.310238    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxjf\" (UniqueName: \"kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.310506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.310486    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.318800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.318777    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"]
Nov 03 03:39:57.366347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.366323    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6e985\""
Nov 03 03:39:57.394771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.394750    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6e985-dockercfg-pkxm7\""
Nov 03 03:39:57.412038 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.412005    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.412130 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.412045    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxjf\" (UniqueName: \"kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.412771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.412752    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.475864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.475836    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxjf\" (UniqueName: \"kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf\") pod \"payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.576360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.576287    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:57.956513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:57.956480    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"]
Nov 03 03:39:57.993028 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:39:57.992994    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe237be_7c05_47df_a59a_c20fb9f33a37.slice/crio-174348a99287d1b4e30b08672da98dc68961414caa577e49b3bcc3f7998afc18 WatchSource:0}: Error finding container 174348a99287d1b4e30b08672da98dc68961414caa577e49b3bcc3f7998afc18: Status 404 returned error can't find the container with id 174348a99287d1b4e30b08672da98dc68961414caa577e49b3bcc3f7998afc18
Nov 03 03:39:58.010763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.010735    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:58.031604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.031578    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"]
Nov 03 03:39:58.038688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.038664    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.094627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.094209    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"]
Nov 03 03:39:58.120603 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.120534    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskjf\" (UniqueName: \"kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf\") pod \"payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.120742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.120721    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.219286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.219263    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:39:58.221588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.221561    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.221769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.221751    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskjf\" (UniqueName: \"kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf\") pod \"payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.222269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.222247    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.283328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.283300    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskjf\" (UniqueName: \"kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf\") pod \"payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.322688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.322660    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.322831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.322815    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.322905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.322880    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.322961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.322913    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.323017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.322958    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.323017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.322997    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.323124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323033    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.323124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323077    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.323242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323111    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:39:58.323242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323150    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs\") pod \"6785670a-a73e-44dd-87b0-5abc41c720b1\" (UID: \"6785670a-a73e-44dd-87b0-5abc41c720b1\") "
Nov 03 03:39:58.323464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323426    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:39:58.323593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323514    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:39:58.323720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323690    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:39:58.323720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323703    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:39:58.323925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323823    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.324069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.323928    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.325185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.325162    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:39:58.325287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.325226    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:58.325513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.325491    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:39:58.325594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.325511    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f" pod "6785670a-a73e-44dd-87b0-5abc41c720b1" (UID: "6785670a-a73e-44dd-87b0-5abc41c720b1"). InnerVolumeSpecName "kube-api-access-kfh6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:39:58.399387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.399356    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:58.425164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425133    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.425164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425162    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6785670a-a73e-44dd-87b0-5abc41c720b1-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.425323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425172    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.425323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425181    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfh6f\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-kube-api-access-kfh6f\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.425323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425191    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.425323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425200    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.425323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.425207    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785670a-a73e-44dd-87b0-5abc41c720b1-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:39:58.468896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.467449    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"]
Nov 03 03:39:58.468896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.467755    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy" containerID="cri-o://da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91" gracePeriod=30
Nov 03 03:39:58.506252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.506225    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:58.786869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.786846    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"]
Nov 03 03:39:58.788285 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:39:58.788262    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cde483c_85f7_4693_85d3_41e47d0700c3.slice/crio-fc7a288d0caaae9bce484022611b660e5dd23dd1cbca21910dd0bb2f32221584 WatchSource:0}: Error finding container fc7a288d0caaae9bce484022611b660e5dd23dd1cbca21910dd0bb2f32221584: Status 404 returned error can't find the container with id fc7a288d0caaae9bce484022611b660e5dd23dd1cbca21910dd0bb2f32221584
Nov 03 03:39:58.939642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:58.939612    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:39:59.029997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.029964    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk" event={"ID":"bbe237be-7c05-47df-a59a-c20fb9f33a37","Type":"ContainerStarted","Data":"b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a"}
Nov 03 03:39:59.030205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.030028    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk" event={"ID":"bbe237be-7c05-47df-a59a-c20fb9f33a37","Type":"ContainerStarted","Data":"174348a99287d1b4e30b08672da98dc68961414caa577e49b3bcc3f7998afc18"}
Nov 03 03:39:59.030313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.030287    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:39:59.031638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.031611    5515 generic.go:356] "Generic (PLEG): container finished" podID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerID="348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1" exitCode=0
Nov 03 03:39:59.031772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.031657    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"
Nov 03 03:39:59.031772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.031695    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" event={"ID":"6785670a-a73e-44dd-87b0-5abc41c720b1","Type":"ContainerDied","Data":"348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1"}
Nov 03 03:39:59.031772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.031745    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77" event={"ID":"6785670a-a73e-44dd-87b0-5abc41c720b1","Type":"ContainerDied","Data":"80fd95c44411dfbdb3da8de9cfb818ce4a7fade8b404c5ff5213683e6299f333"}
Nov 03 03:39:59.031955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.031773    5515 scope.go:122] "RemoveContainer" containerID="348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1"
Nov 03 03:39:59.033605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.033579    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7" event={"ID":"0cde483c-85f7-4693-85d3-41e47d0700c3","Type":"ContainerStarted","Data":"c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753"}
Nov 03 03:39:59.033751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.033629    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7" event={"ID":"0cde483c-85f7-4693-85d3-41e47d0700c3","Type":"ContainerStarted","Data":"fc7a288d0caaae9bce484022611b660e5dd23dd1cbca21910dd0bb2f32221584"}
Nov 03 03:39:59.033751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.033657    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:39:59.041137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.041119    5515 scope.go:122] "RemoveContainer" containerID="348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1"
Nov 03 03:39:59.041399 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:39:59.041378    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1\": container with ID starting with 348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1 not found: ID does not exist" containerID="348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1"
Nov 03 03:39:59.041448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.041404    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1"} err="failed to get container status \"348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1\": rpc error: code = NotFound desc = could not find container \"348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1\": container with ID starting with 348f1a9e65f5a3b43401e0c3182554cfe28a1d6ed108201dff0435ad919de1a1 not found: ID does not exist"
Nov 03 03:39:59.153942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.153885    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk" podStartSLOduration=2.153868126 podStartE2EDuration="2.153868126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:39:59.150275899 +0000 UTC m=+2290.381168319" watchObservedRunningTime="2026-11-03 03:39:59.153868126 +0000 UTC m=+2290.384760543"
Nov 03 03:39:59.293571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.293471    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"]
Nov 03 03:39:59.301401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.298945    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-dpj77"]
Nov 03 03:39:59.393097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:39:59.393048    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7" podStartSLOduration=2.393032919 podStartE2EDuration="2.393032919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:39:59.392842014 +0000 UTC m=+2290.623734433" watchObservedRunningTime="2026-11-03 03:39:59.393032919 +0000 UTC m=+2290.623925336"
Nov 03 03:40:00.114664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.114629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:00.530236 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.530157    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"]
Nov 03 03:40:00.530771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.530756    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy"
Nov 03 03:40:00.530830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.530772    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy"
Nov 03 03:40:00.530868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.530858    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" containerName="istio-proxy"
Nov 03 03:40:00.535220 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.535190    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.589399 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.589365    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-754aeb-openshift-default-dockercfg-gswtg\""
Nov 03 03:40:00.636526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.636496    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.655851    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.655947    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656008    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656104    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656173    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654zd\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656220    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656267    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656314    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.657421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.656356    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.749601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.749081    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"]
Nov 03 03:40:00.758226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.758194    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.758538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.758521    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.758709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.758694    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.758851 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.758837    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.758885    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759039    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759224 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759066    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759333 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759314    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654zd\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759440    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759589    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759621    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759667    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.759891 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.759764    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.760316 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.760289    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.762291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.762266    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.762795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.762771    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.791154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.791087    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:00.820494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.820462    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.820690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.820669    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654zd\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd\") pod \"e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") " pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.851252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.851228    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:00.926813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.926779    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6785670a-a73e-44dd-87b0-5abc41c720b1" path="/var/lib/kubelet/pods/6785670a-a73e-44dd-87b0-5abc41c720b1/volumes"
Nov 03 03:40:00.943890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:00.943854    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:01.311080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:01.311058    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"]
Nov 03 03:40:01.312672 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:01.312640    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa7c9b5_972c_4d09_b22d_b60f725f1f8c.slice/crio-dcd771e1ed356de3010ddfe2e2aa41a797920defe37a49464557a20a03b1bbae WatchSource:0}: Error finding container dcd771e1ed356de3010ddfe2e2aa41a797920defe37a49464557a20a03b1bbae: Status 404 returned error can't find the container with id dcd771e1ed356de3010ddfe2e2aa41a797920defe37a49464557a20a03b1bbae
Nov 03 03:40:01.315215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:01.315179    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:40:01.315317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:01.315249    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:40:01.315317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:01.315279    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:40:02.053175 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:02.053131    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" event={"ID":"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c","Type":"ContainerStarted","Data":"f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"}
Nov 03 03:40:02.053624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:02.053199    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" event={"ID":"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c","Type":"ContainerStarted","Data":"dcd771e1ed356de3010ddfe2e2aa41a797920defe37a49464557a20a03b1bbae"}
Nov 03 03:40:02.230258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:02.230204    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podStartSLOduration=2.230191664 podStartE2EDuration="2.230191664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:02.227648759 +0000 UTC m=+2293.458541228" watchObservedRunningTime="2026-11-03 03:40:02.230191664 +0000 UTC m=+2293.461084081"
Nov 03 03:40:02.852232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:02.852201    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:02.857725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:02.857703    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:03.059900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.059867    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:03.061062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.061035    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:40:03.433103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.433069    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"]
Nov 03 03:40:03.433288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.433269    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" podUID="04fb8e4e-456f-4f30-b902-81444f1030be" containerName="maas-api" containerID="cri-o://0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196" gracePeriod=30
Nov 03 03:40:03.536357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.536310    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:03.764132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.764091    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:40:03.810122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.810098    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\") pod \"04fb8e4e-456f-4f30-b902-81444f1030be\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") "
Nov 03 03:40:03.810289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.810153    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca\") pod \"04fb8e4e-456f-4f30-b902-81444f1030be\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") "
Nov 03 03:40:03.810289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.810231    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs\") pod \"04fb8e4e-456f-4f30-b902-81444f1030be\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") "
Nov 03 03:40:03.810289 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.810289    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns\" (UniqueName: \"kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns\") pod \"04fb8e4e-456f-4f30-b902-81444f1030be\" (UID: \"04fb8e4e-456f-4f30-b902-81444f1030be\") "
Nov 03 03:40:03.810596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.810529    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca" pod "04fb8e4e-456f-4f30-b902-81444f1030be" (UID: "04fb8e4e-456f-4f30-b902-81444f1030be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:40:03.810881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.810746    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04fb8e4e-456f-4f30-b902-81444f1030be-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:03.811156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.811140    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:40:03.812274 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.812243    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls" pod "04fb8e4e-456f-4f30-b902-81444f1030be" (UID: "04fb8e4e-456f-4f30-b902-81444f1030be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:40:03.812365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.812311    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs" pod "04fb8e4e-456f-4f30-b902-81444f1030be" (UID: "04fb8e4e-456f-4f30-b902-81444f1030be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:40:03.812457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.812436    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns" pod "04fb8e4e-456f-4f30-b902-81444f1030be" (UID: "04fb8e4e-456f-4f30-b902-81444f1030be"). InnerVolumeSpecName "kube-api-access-md4ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912094    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912213    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912251    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912284    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912337    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912381    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912408    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912496    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.912648    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql\") pod \"5c240ad0-f62f-4516-8824-c710d0e960c9\" (UID: \"5c240ad0-f62f-4516-8824-c710d0e960c9\") "
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.913117    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.913141    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4ns\" (UniqueName: \"kubernetes.io/projected/04fb8e4e-456f-4f30-b902-81444f1030be-kube-api-access-md4ns\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.913158    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04fb8e4e-456f-4f30-b902-81444f1030be-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.924491    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.924730    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.924896    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.925147    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.925208    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:03.929628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.925725    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:03.933711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.932536    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:40:03.940100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.934917    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:03.940100 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:03.935072    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql" pod "5c240ad0-f62f-4516-8824-c710d0e960c9" (UID: "5c240ad0-f62f-4516-8824-c710d0e960c9"). InnerVolumeSpecName "kube-api-access-ql5ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:04.014868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014774    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.014868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014811    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.014868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014826    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.014868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014839    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.014868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014853    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.014868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014866    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.015287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014879    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c240ad0-f62f-4516-8824-c710d0e960c9-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.015287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014894    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5ql\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-kube-api-access-ql5ql\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.015287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.014908    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c240ad0-f62f-4516-8824-c710d0e960c9-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.076541    5515 generic.go:356] "Generic (PLEG): container finished" podID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerID="da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91" exitCode=0
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.076620    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.076612    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" event={"ID":"5c240ad0-f62f-4516-8824-c710d0e960c9","Type":"ContainerDied","Data":"da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91"}
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.076783    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d" event={"ID":"5c240ad0-f62f-4516-8824-c710d0e960c9","Type":"ContainerDied","Data":"c642e944caf33bbb735af4158b1505bd10e3be10cb91d3e390151efe16e7da1a"}
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.076825    5515 scope.go:122] "RemoveContainer" containerID="da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91"
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.078271    5515 generic.go:356] "Generic (PLEG): container finished" podID="04fb8e4e-456f-4f30-b902-81444f1030be" containerID="0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196" exitCode=0
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.078450    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" event={"ID":"04fb8e4e-456f-4f30-b902-81444f1030be","Type":"ContainerDied","Data":"0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196"}
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.078499    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j" event={"ID":"04fb8e4e-456f-4f30-b902-81444f1030be","Type":"ContainerDied","Data":"28c5feac7b4cecb9a8200e56558c7e0e15a2fa38d89252835f4d4cc602a43002"}
Nov 03 03:40:04.082589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.078523    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"
Nov 03 03:40:04.090182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.090160    5515 scope.go:122] "RemoveContainer" containerID="fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"
Nov 03 03:40:04.108642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.108620    5515 scope.go:122] "RemoveContainer" containerID="da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91"
Nov 03 03:40:04.108900 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:04.108877    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91\": container with ID starting with da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91 not found: ID does not exist" containerID="da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91"
Nov 03 03:40:04.108971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.108911    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91"} err="failed to get container status \"da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91\": rpc error: code = NotFound desc = could not find container \"da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91\": container with ID starting with da192caf5472ee092362de2361e549e2efb60b64b92b5cf8496cf81574a7be91 not found: ID does not exist"
Nov 03 03:40:04.108971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.108935    5515 scope.go:122] "RemoveContainer" containerID="fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"
Nov 03 03:40:04.109189 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:04.109171    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17\": container with ID starting with fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17 not found: ID does not exist" containerID="fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"
Nov 03 03:40:04.109233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.109196    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17"} err="failed to get container status \"fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17\": rpc error: code = NotFound desc = could not find container \"fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17\": container with ID starting with fea771123e141a0879d4a34948584a37fc119bb1b6dd5337ead1a68342b65d17 not found: ID does not exist"
Nov 03 03:40:04.109233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.109210    5515 scope.go:122] "RemoveContainer" containerID="0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196"
Nov 03 03:40:04.118240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.118221    5515 scope.go:122] "RemoveContainer" containerID="0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196"
Nov 03 03:40:04.118478 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:04.118461    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196\": container with ID starting with 0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196 not found: ID does not exist" containerID="0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196"
Nov 03 03:40:04.118530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.118486    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196"} err="failed to get container status \"0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196\": rpc error: code = NotFound desc = could not find container \"0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196\": container with ID starting with 0fcdbf3de4d80c8fed2cec2571877e1d0dc1f4d94700fd5efe6109485ba68196 not found: ID does not exist"
Nov 03 03:40:04.122341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.122318    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:04.246465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.246432    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"]
Nov 03 03:40:04.250375 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.250352    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f1c469-openshift-default-5bbbc765c9-w2k2d"]
Nov 03 03:40:04.272836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.272777    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:04.374855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.374822    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"]
Nov 03 03:40:04.378074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.377920    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-160d39-86ddbb66dd-6h49j"]
Nov 03 03:40:04.893940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.893899    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fb8e4e-456f-4f30-b902-81444f1030be" path="/var/lib/kubelet/pods/04fb8e4e-456f-4f30-b902-81444f1030be/volumes"
Nov 03 03:40:04.894439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:04.894417    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" path="/var/lib/kubelet/pods/5c240ad0-f62f-4516-8824-c710d0e960c9/volumes"
Nov 03 03:40:05.006755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:05.006727    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:05.203715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:05.203640    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:05.351800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:05.351770    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:05.786624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:05.786592    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:08.228320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.228290    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:08.409825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.409785    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:08.643356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643325    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"]
Nov 03 03:40:08.643893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643879    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy"
Nov 03 03:40:08.643949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643893    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy"
Nov 03 03:40:08.643949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643907    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy"
Nov 03 03:40:08.643949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643912    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy"
Nov 03 03:40:08.643949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643934    5515 cpu_manager.go:414] "containerMap: removing container" podUID="04fb8e4e-456f-4f30-b902-81444f1030be" containerName="maas-api"
Nov 03 03:40:08.643949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.643939    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fb8e4e-456f-4f30-b902-81444f1030be" containerName="maas-api"
Nov 03 03:40:08.644109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.644019    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy"
Nov 03 03:40:08.644109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.644033    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fb8e4e-456f-4f30-b902-81444f1030be" containerName="maas-api"
Nov 03 03:40:08.647247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.647232    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.680718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.680692    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmbq\" (UniqueName: \"kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq\") pod \"payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.680855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.680733    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.702163 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.702136    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-754aeb\""
Nov 03 03:40:08.705096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.705070    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"]
Nov 03 03:40:08.796572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.792817    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.796572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.793097    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmbq\" (UniqueName: \"kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq\") pod \"payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.796572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.796406    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.821578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.821172    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"]
Nov 03 03:40:08.825572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.822157    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c240ad0-f62f-4516-8824-c710d0e960c9" containerName="istio-proxy"
Nov 03 03:40:08.828007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.827944    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:08.837215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.832857    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:08.855826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.855789    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmbq\" (UniqueName: \"kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq\") pod \"payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.873006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.870470    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"]
Nov 03 03:40:08.891578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.888486    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-754aeb-dockercfg-w8fzc\""
Nov 03 03:40:08.908622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.905784    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:08.908622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.905892    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxwm\" (UniqueName: \"kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm\") pod \"payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:08.960896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.960868    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:08.978912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:08.978524    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:09.006625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.006595    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxwm\" (UniqueName: \"kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm\") pod \"payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:09.006791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.006771    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:09.007456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.007428    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:09.068794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.068762    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxwm\" (UniqueName: \"kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm\") pod \"payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:09.201527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.201445    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:09.338524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.338500    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"]
Nov 03 03:40:09.340110 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:09.340084    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0d3818_5409_4e51_ab2f_296ae66a954c.slice/crio-2dd392f69c63f3ea5d74fb9b72bfff11527b16f36640fb231b083155f4cf4196 WatchSource:0}: Error finding container 2dd392f69c63f3ea5d74fb9b72bfff11527b16f36640fb231b083155f4cf4196: Status 404 returned error can't find the container with id 2dd392f69c63f3ea5d74fb9b72bfff11527b16f36640fb231b083155f4cf4196
Nov 03 03:40:09.577668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:09.577645    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"]
Nov 03 03:40:09.578643 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:09.578615    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734b45c9_9083_4105_890a_3a9f18a6494f.slice/crio-cb70d34b7a49fb7036f5b55bf96d414ea947e7243afc697d2122d4dcad7e0627 WatchSource:0}: Error finding container cb70d34b7a49fb7036f5b55bf96d414ea947e7243afc697d2122d4dcad7e0627: Status 404 returned error can't find the container with id cb70d34b7a49fb7036f5b55bf96d414ea947e7243afc697d2122d4dcad7e0627
Nov 03 03:40:10.042735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.042699    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:40:10.043124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.043101    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:40:10.124348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.124321    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv" event={"ID":"0c0d3818-5409-4e51-ab2f-296ae66a954c","Type":"ContainerStarted","Data":"952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b"}
Nov 03 03:40:10.124525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.124373    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:10.124525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.124393    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv" event={"ID":"0c0d3818-5409-4e51-ab2f-296ae66a954c","Type":"ContainerStarted","Data":"2dd392f69c63f3ea5d74fb9b72bfff11527b16f36640fb231b083155f4cf4196"}
Nov 03 03:40:10.125810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.125788    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs" event={"ID":"734b45c9-9083-4105-890a-3a9f18a6494f","Type":"ContainerStarted","Data":"628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50"}
Nov 03 03:40:10.125913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.125824    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs" event={"ID":"734b45c9-9083-4105-890a-3a9f18a6494f","Type":"ContainerStarted","Data":"cb70d34b7a49fb7036f5b55bf96d414ea947e7243afc697d2122d4dcad7e0627"}
Nov 03 03:40:10.125913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.125851    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:10.347713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.347633    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:10.405145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.404236    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs" podStartSLOduration=2.404217461 podStartE2EDuration="2.404217461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:10.403612934 +0000 UTC m=+2301.634505352" watchObservedRunningTime="2026-11-03 03:40:10.404217461 +0000 UTC m=+2301.635109881"
Nov 03 03:40:10.524575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.524491    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv" podStartSLOduration=2.524472315 podStartE2EDuration="2.524472315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:10.524060866 +0000 UTC m=+2301.754953289" watchObservedRunningTime="2026-11-03 03:40:10.524472315 +0000 UTC m=+2301.755364733"
Nov 03 03:40:10.807070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.807038    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"]
Nov 03 03:40:10.807343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.807313    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy" containerID="cri-o://2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5" gracePeriod=30
Nov 03 03:40:10.840721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:10.840699    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:11.628728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:11.628697    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:13.126216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:13.126187    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:15.494969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:15.494938    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:16.153235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.153211    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:40:16.163688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.163662    5515 generic.go:356] "Generic (PLEG): container finished" podID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerID="2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5" exitCode=0
Nov 03 03:40:16.163831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.163709    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"
Nov 03 03:40:16.163831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.163746    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" event={"ID":"f46f4684-5004-4dfe-aaaf-bb878fbf0e57","Type":"ContainerDied","Data":"2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5"}
Nov 03 03:40:16.163959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.163840    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8" event={"ID":"f46f4684-5004-4dfe-aaaf-bb878fbf0e57","Type":"ContainerDied","Data":"396926b7dbd03ba81aea7049078d065d2a9cf28e7f201c10faa65d760658d5cc"}
Nov 03 03:40:16.163959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.163874    5515 scope.go:122] "RemoveContainer" containerID="2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5"
Nov 03 03:40:16.173359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.173343    5515 scope.go:122] "RemoveContainer" containerID="efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"
Nov 03 03:40:16.182507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.182491    5515 scope.go:122] "RemoveContainer" containerID="2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5"
Nov 03 03:40:16.182813 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:16.182793    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5\": container with ID starting with 2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5 not found: ID does not exist" containerID="2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5"
Nov 03 03:40:16.182912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.182822    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5"} err="failed to get container status \"2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5\": rpc error: code = NotFound desc = could not find container \"2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5\": container with ID starting with 2d4eaf319e832691954c73a37e6917c0bef4f88e1e2744af67713b5b0d6a0da5 not found: ID does not exist"
Nov 03 03:40:16.182912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.182840    5515 scope.go:122] "RemoveContainer" containerID="efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"
Nov 03 03:40:16.183088 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:16.183066    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2\": container with ID starting with efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2 not found: ID does not exist" containerID="efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"
Nov 03 03:40:16.183129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.183095    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2"} err="failed to get container status \"efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2\": rpc error: code = NotFound desc = could not find container \"efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2\": container with ID starting with efc18e01234b3758e24deb21fba0709126776d25bf1007522941141bda41f7e2 not found: ID does not exist"
Nov 03 03:40:16.231879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.231856    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.231981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.231951    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.231982    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232004    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232074    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232105    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:16.232189 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232159    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232197    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:16.232245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232207    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232239    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232268    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert\") pod \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\" (UID: \"f46f4684-5004-4dfe-aaaf-bb878fbf0e57\") "
Nov 03 03:40:16.232401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232336    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:16.232521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232418    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:16.232872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232803    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.232872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232831    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.232872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232790    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:40:16.232872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232872    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.232872 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.232902    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.234238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.234218    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:40:16.234444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.234426    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "kube-api-access-wnlbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:16.234838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.234813    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:16.235134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.235106    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token" pod "f46f4684-5004-4dfe-aaaf-bb878fbf0e57" (UID: "f46f4684-5004-4dfe-aaaf-bb878fbf0e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:16.265140 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.265110    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"]
Nov 03 03:40:16.265677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.265664    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy"
Nov 03 03:40:16.265726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.265678    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy"
Nov 03 03:40:16.265726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.265705    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy"
Nov 03 03:40:16.265726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.265710    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy"
Nov 03 03:40:16.265821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.265797    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy"
Nov 03 03:40:16.270406 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.270390    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334056    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334190    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334325    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdf5\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334398    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334436    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334515    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334541    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334689    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334721    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.334877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334861    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnlbj\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-kube-api-access-wnlbj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.334947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334882    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.334947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334893    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.334947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334904    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.334947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.334913    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46f4684-5004-4dfe-aaaf-bb878fbf0e57-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:16.363379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.363347    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:16.376705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.376685    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-545809-openshift-default-dockercfg-sjfjw\""
Nov 03 03:40:16.379952 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.379892    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"]
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441523    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441606    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441697    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441826    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdf5\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441904    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441947    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.441978    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.442010    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.443571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.442085    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.452581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.451991    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.452581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.452317    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.455013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.454013    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.455013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.454147    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.455013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.454260    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.455013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.455613    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.455013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.460070    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.515630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.515603    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.515946 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.515855    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdf5\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5\") pod \"e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") " pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.567145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.567095    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:16.584021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.583984    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:16.628060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.627972    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"]
Nov 03 03:40:16.631416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.631391    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-160d39-openshift-default-5466876d49-8xbw8"]
Nov 03 03:40:16.893619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:16.893508    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" path="/var/lib/kubelet/pods/f46f4684-5004-4dfe-aaaf-bb878fbf0e57/volumes"
Nov 03 03:40:17.014439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.014415    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"]
Nov 03 03:40:17.015370 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:17.015344    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3679b6_68f5_4844_9785_4e0f69ff9c79.slice/crio-b433779d5878339117706532954ecb2ff987eedd60de5922c9299ed357e2e8e1 WatchSource:0}: Error finding container b433779d5878339117706532954ecb2ff987eedd60de5922c9299ed357e2e8e1: Status 404 returned error can't find the container with id b433779d5878339117706532954ecb2ff987eedd60de5922c9299ed357e2e8e1
Nov 03 03:40:17.017677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.017647    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:40:17.017761 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.017705    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:40:17.017761 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.017735    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:40:17.171230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.171140    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" event={"ID":"5a3679b6-68f5-4844-9785-4e0f69ff9c79","Type":"ContainerStarted","Data":"ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"}
Nov 03 03:40:17.171230 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.171203    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" event={"ID":"5a3679b6-68f5-4844-9785-4e0f69ff9c79","Type":"ContainerStarted","Data":"b433779d5878339117706532954ecb2ff987eedd60de5922c9299ed357e2e8e1"}
Nov 03 03:40:17.294415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.294361    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podStartSLOduration=1.294347787 podStartE2EDuration="1.294347787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:17.293750301 +0000 UTC m=+2308.524642720" watchObservedRunningTime="2026-11-03 03:40:17.294347787 +0000 UTC m=+2308.525240201"
Nov 03 03:40:17.584779 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:17.584700    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:18.078136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:18.078103    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:18.505315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:18.505284    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:18.590671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:18.590642    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:19.193521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:19.193492    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:19.194348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:19.194329    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:40:20.072860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:20.072826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:20.984905 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:20.984863    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:40:21.134031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:21.134002    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:40:21.134493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:21.134201    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:40:21.860877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:21.860846    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:22.428703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:22.428674    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:22.992930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:22.992886    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:23.377158 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:23.377112    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:23.576226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:23.567106    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:23.927416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:23.927384    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"]
Nov 03 03:40:23.930638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:23.928356    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46f4684-5004-4dfe-aaaf-bb878fbf0e57" containerName="istio-proxy"
Nov 03 03:40:23.960661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:23.960630    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.039725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.022762    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"]
Nov 03 03:40:24.043580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.041142    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-545809-dockercfg-jk2c4\""
Nov 03 03:40:24.043580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.041234    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-545809\""
Nov 03 03:40:24.053080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.053048    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsv5d\" (UniqueName: \"kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d\") pod \"payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.053424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.053398    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.098235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.098211    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"]
Nov 03 03:40:24.102318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.102299    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.118408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.118387    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"]
Nov 03 03:40:24.122172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.122153    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.155094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155071    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.155210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155111    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.155255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155200    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdc4d\" (UniqueName: \"kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d\") pod \"payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.155332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155280    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdn42\" (UniqueName: \"kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.155332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155312    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.155533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155341    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv5d\" (UniqueName: \"kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d\") pod \"payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.156075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.155980    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.156676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.156651    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8200d4-dockercfg-mt2n6\""
Nov 03 03:40:24.156896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.156877    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8200d4\""
Nov 03 03:40:24.159439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.159411    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"]
Nov 03 03:40:24.181539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.181479    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"]
Nov 03 03:40:24.217935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.217912    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv5d\" (UniqueName: \"kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d\") pod \"payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.256589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.256528    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.256722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.256650    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdc4d\" (UniqueName: \"kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d\") pod \"payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.256722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.256696    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdn42\" (UniqueName: \"kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.256794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.256723    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.257256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.257238    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.257410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.257390    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.319014 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.318987    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdc4d\" (UniqueName: \"kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d\") pod \"payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.319144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.319071    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdn42\" (UniqueName: \"kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42\") pod \"payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.346291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.346268    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:24.415897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.415867    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:24.434916 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.434889    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:24.724337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.724248    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"]
Nov 03 03:40:24.728798 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:24.728748    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0e5a39_b560_4e8e_a549_c84e1f6339a1.slice/crio-54744bea9b2213c70b0b11e33a2cd1ed15b6e7763bf1b5627cbdc88ebb74ace4 WatchSource:0}: Error finding container 54744bea9b2213c70b0b11e33a2cd1ed15b6e7763bf1b5627cbdc88ebb74ace4: Status 404 returned error can't find the container with id 54744bea9b2213c70b0b11e33a2cd1ed15b6e7763bf1b5627cbdc88ebb74ace4
Nov 03 03:40:24.803605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.803542    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"]
Nov 03 03:40:24.805602 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:24.805537    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f95be54_2b87_4ade_970e_0a93760a7da9.slice/crio-46de732b46846f9068e6a8ca8603fa75280d98ecacc720bde14c060c02be8080 WatchSource:0}: Error finding container 46de732b46846f9068e6a8ca8603fa75280d98ecacc720bde14c060c02be8080: Status 404 returned error can't find the container with id 46de732b46846f9068e6a8ca8603fa75280d98ecacc720bde14c060c02be8080
Nov 03 03:40:24.808161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.808144    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"]
Nov 03 03:40:24.808246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.808226    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:24.832975 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.832954    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"]
Nov 03 03:40:24.835857 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:24.835833    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76deac35_77c1_4ab9_9ea8_8de8e48a5e22.slice/crio-5217c795c6fd66c862493f06da6d78abdeaa6d62772f305655d68cf6e9e4ded9 WatchSource:0}: Error finding container 5217c795c6fd66c862493f06da6d78abdeaa6d62772f305655d68cf6e9e4ded9: Status 404 returned error can't find the container with id 5217c795c6fd66c862493f06da6d78abdeaa6d62772f305655d68cf6e9e4ded9
Nov 03 03:40:24.866845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.865509    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"]
Nov 03 03:40:24.866845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.866393    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9g76\" (UniqueName: \"kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76\") pod \"payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:24.866845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.866447    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:24.968118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.968088    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9g76\" (UniqueName: \"kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76\") pod \"payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:24.968268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.968139    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:24.968875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:24.968854    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:25.030277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.030219    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9g76\" (UniqueName: \"kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76\") pod \"payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:25.124709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.124687    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:25.254174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.254135    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq" event={"ID":"4a0e5a39-b560-4e8e-a549-c84e1f6339a1","Type":"ContainerStarted","Data":"4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0"}
Nov 03 03:40:25.254350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.254192    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq" event={"ID":"4a0e5a39-b560-4e8e-a549-c84e1f6339a1","Type":"ContainerStarted","Data":"54744bea9b2213c70b0b11e33a2cd1ed15b6e7763bf1b5627cbdc88ebb74ace4"}
Nov 03 03:40:25.254350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.254220    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:25.255869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.255844    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf" event={"ID":"76deac35-77c1-4ab9-9ea8-8de8e48a5e22","Type":"ContainerStarted","Data":"777d1aca1faedd06140cee6f21026e688382890fa3bcfb7ac2e2439e02615b98"}
Nov 03 03:40:25.255996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.255893    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf" event={"ID":"76deac35-77c1-4ab9-9ea8-8de8e48a5e22","Type":"ContainerStarted","Data":"5217c795c6fd66c862493f06da6d78abdeaa6d62772f305655d68cf6e9e4ded9"}
Nov 03 03:40:25.255996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.255934    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:25.257426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.257403    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk" event={"ID":"7f95be54-2b87-4ade-970e-0a93760a7da9","Type":"ContainerStarted","Data":"19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0"}
Nov 03 03:40:25.257534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.257438    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk" event={"ID":"7f95be54-2b87-4ade-970e-0a93760a7da9","Type":"ContainerStarted","Data":"46de732b46846f9068e6a8ca8603fa75280d98ecacc720bde14c060c02be8080"}
Nov 03 03:40:25.257534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.257467    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:25.374070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.374015    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq" podStartSLOduration=2.373997776 podStartE2EDuration="2.373997776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:25.372255526 +0000 UTC m=+2316.603147955" watchObservedRunningTime="2026-11-03 03:40:25.373997776 +0000 UTC m=+2316.604890194"
Nov 03 03:40:25.510615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.510521    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf" podStartSLOduration=1.510502211 podStartE2EDuration="1.510502211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:25.509892173 +0000 UTC m=+2316.740784591" watchObservedRunningTime="2026-11-03 03:40:25.510502211 +0000 UTC m=+2316.741394629"
Nov 03 03:40:25.524671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.524643    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"]
Nov 03 03:40:25.653489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.653366    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk" podStartSLOduration=1.6533434009999999 podStartE2EDuration="1.653343401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:25.632520679 +0000 UTC m=+2316.863413104" watchObservedRunningTime="2026-11-03 03:40:25.653343401 +0000 UTC m=+2316.884235821"
Nov 03 03:40:25.799051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:25.799018    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:26.264676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:26.264642    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t" event={"ID":"00e9907a-a6e2-4abf-8311-40fdba1cf98e","Type":"ContainerStarted","Data":"90de5a6a3e90bd6e4b0b893b2ea02bf2bff27206fce726ec98e10ed8af37d4d6"}
Nov 03 03:40:26.264893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:26.264696    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t" event={"ID":"00e9907a-a6e2-4abf-8311-40fdba1cf98e","Type":"ContainerStarted","Data":"a9e5552390b7b2b2345c9db7691f013783d2f27ccf80a58d35f04869861331bb"}
Nov 03 03:40:26.265182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:26.265141    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:26.386180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:26.386119    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t" podStartSLOduration=2.38610239 podStartE2EDuration="2.38610239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:26.3835217 +0000 UTC m=+2317.614414119" watchObservedRunningTime="2026-11-03 03:40:26.38610239 +0000 UTC m=+2317.616994813"
Nov 03 03:40:28.000927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:28.000888    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:40:28.821531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:28.821493    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:30.834311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:30.834278    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:32.787709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:32.787677    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:32.938381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:32.938345    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:33.086462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:33.086381    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:36.020410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:36.020381    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:36.267784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:36.267754    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:40:36.268064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:36.268036    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:40:36.268931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:36.268911    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:40:38.278609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:38.278567    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:40:40.407688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:40.407648    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:40.927924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:40.927888    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:41.072055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:41.072015    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:42.230503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:42.230468    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:46.557425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:46.557390    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:46.704676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:46.704643    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:47.180862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:47.180825    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:48.145991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:48.145957    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:48.641641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:48.641609    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"]
Nov 03 03:40:48.641869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:48.641846    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy" containerID="cri-o://fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce" gracePeriod=30
Nov 03 03:40:48.648383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:48.648361    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:50.172233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:50.172161    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:50.858165 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:50.858134    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:50.996534 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:50.996491    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:40:51.172540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:51.172366    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:52.920094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:52.920005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:53.907904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.907854    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:40:53.942602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942575    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942662    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942718    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942752    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942817    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942856    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942867    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942895    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942922    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.942965    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token\") pod \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\" (UID: \"9dcad0ae-8c13-431c-a3f1-ac2733e2b488\") "
Nov 03 03:40:53.943474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943037    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:53.943474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943077    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:53.943474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943107    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:53.943474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943202    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:40:53.943732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943526    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:53.943732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943575    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:53.943732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943592    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:53.943732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943608    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:53.943732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.943637    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:53.945106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.945083    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:40:53.945275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.945260    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:53.945521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.945508    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:40:53.945664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:53.945646    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48" pod "9dcad0ae-8c13-431c-a3f1-ac2733e2b488" (UID: "9dcad0ae-8c13-431c-a3f1-ac2733e2b488"). InnerVolumeSpecName "kube-api-access-lwx48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:40:54.044517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.044495    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:54.044517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.044517    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwx48\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-kube-api-access-lwx48\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:54.044678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.044527    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:54.044678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.044537    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dcad0ae-8c13-431c-a3f1-ac2733e2b488-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:40:54.449180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.449144    5515 generic.go:356] "Generic (PLEG): container finished" podID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerID="fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce" exitCode=0
Nov 03 03:40:54.449348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.449196    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"
Nov 03 03:40:54.449348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.449223    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" event={"ID":"9dcad0ae-8c13-431c-a3f1-ac2733e2b488","Type":"ContainerDied","Data":"fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce"}
Nov 03 03:40:54.449348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.449276    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn" event={"ID":"9dcad0ae-8c13-431c-a3f1-ac2733e2b488","Type":"ContainerDied","Data":"ef839ab0eb81395de7e1b4984b4e2c3a52bf3ee3eec9e895aee5ab9e43350bf8"}
Nov 03 03:40:54.449348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.449297    5515 scope.go:122] "RemoveContainer" containerID="fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce"
Nov 03 03:40:54.459997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.459979    5515 scope.go:122] "RemoveContainer" containerID="c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"
Nov 03 03:40:54.471284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.471264    5515 scope.go:122] "RemoveContainer" containerID="fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce"
Nov 03 03:40:54.471596 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:54.471577    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce\": container with ID starting with fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce not found: ID does not exist" containerID="fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce"
Nov 03 03:40:54.471681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.471602    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce"} err="failed to get container status \"fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce\": rpc error: code = NotFound desc = could not find container \"fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce\": container with ID starting with fe17eda649aec3676c60e64a443f44925e2b0e32f3daaaa37d04e08588ab32ce not found: ID does not exist"
Nov 03 03:40:54.471681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.471619    5515 scope.go:122] "RemoveContainer" containerID="c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"
Nov 03 03:40:54.471880 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:40:54.471864    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e\": container with ID starting with c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e not found: ID does not exist" containerID="c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"
Nov 03 03:40:54.471921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.471886    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e"} err="failed to get container status \"c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e\": rpc error: code = NotFound desc = could not find container \"c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e\": container with ID starting with c7ab12c89e860c8760ff7721f2530a7b77c88a3b4e3f8faaf86a21c43581a08e not found: ID does not exist"
Nov 03 03:40:54.618848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.618815    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"]
Nov 03 03:40:54.669768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.669733    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6ef2c-openshift-default-645885c9fc-gtdpn"]
Nov 03 03:40:54.803492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.803385    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:54.894119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.894087    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" path="/var/lib/kubelet/pods/9dcad0ae-8c13-431c-a3f1-ac2733e2b488/volumes"
Nov 03 03:40:54.974727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:54.974701    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:55.124611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:55.124567    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:58.058995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.058951    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:58.229832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.229797    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:40:58.475786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.475754    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"]
Nov 03 03:40:58.476361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.476347    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy"
Nov 03 03:40:58.476409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.476363    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy"
Nov 03 03:40:58.476409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.476373    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy"
Nov 03 03:40:58.476409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.476377    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy"
Nov 03 03:40:58.476509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.476460    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy"
Nov 03 03:40:58.476509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.476468    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcad0ae-8c13-431c-a3f1-ac2733e2b488" containerName="istio-proxy"
Nov 03 03:40:58.481435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.481417    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.538586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.538536    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"]
Nov 03 03:40:58.615711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.597852    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b755ee\""
Nov 03 03:40:58.629197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.618673    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286wq\" (UniqueName: \"kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq\") pod \"payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.629197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.618752    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.629197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.619099    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b755ee-dockercfg-rxwdp\""
Nov 03 03:40:58.643930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.643889    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"]
Nov 03 03:40:58.652823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.652791    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.690182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.690149    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"]
Nov 03 03:40:58.719804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.719775    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286wq\" (UniqueName: \"kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq\") pod \"payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.719973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.719829    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.719973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.719867    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dmj\" (UniqueName: \"kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj\") pod \"payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.719973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.719968    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.720587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.720569    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.782344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.782270    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-286wq\" (UniqueName: \"kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq\") pod \"payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.797049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.797024    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:58.821473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.821445    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.821615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.821589    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dmj\" (UniqueName: \"kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj\") pod \"payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.822381 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.822357    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.889610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.889516    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dmj\" (UniqueName: \"kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj\") pod \"payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:58.970281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:58.970248    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:59.173021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.172993    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"]
Nov 03 03:40:59.173923 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:59.173897    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1baa566_ebed_4a3d_8232_c6cbd115d30e.slice/crio-9590d2b6ec035ec90f2cccb1bc385f959b9d23c668eb65b6dfb4eeb43230b6e7 WatchSource:0}: Error finding container 9590d2b6ec035ec90f2cccb1bc385f959b9d23c668eb65b6dfb4eeb43230b6e7: Status 404 returned error can't find the container with id 9590d2b6ec035ec90f2cccb1bc385f959b9d23c668eb65b6dfb4eeb43230b6e7
Nov 03 03:40:59.345678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.345654    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"]
Nov 03 03:40:59.346936 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:40:59.346909    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc653fd77_8aa7_4af3_a244_80a1a8c135c2.slice/crio-a50e184a1ff71dcb63309f38aed6ccb67c2fe1b6880369294bb33e10173b2d32 WatchSource:0}: Error finding container a50e184a1ff71dcb63309f38aed6ccb67c2fe1b6880369294bb33e10173b2d32: Status 404 returned error can't find the container with id a50e184a1ff71dcb63309f38aed6ccb67c2fe1b6880369294bb33e10173b2d32
Nov 03 03:40:59.482592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.482488    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m" event={"ID":"c653fd77-8aa7-4af3-a244-80a1a8c135c2","Type":"ContainerStarted","Data":"ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e"}
Nov 03 03:40:59.482592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.482579    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m" event={"ID":"c653fd77-8aa7-4af3-a244-80a1a8c135c2","Type":"ContainerStarted","Data":"a50e184a1ff71dcb63309f38aed6ccb67c2fe1b6880369294bb33e10173b2d32"}
Nov 03 03:40:59.482771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.482735    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:40:59.484207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.484185    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn" event={"ID":"a1baa566-ebed-4a3d-8232-c6cbd115d30e","Type":"ContainerStarted","Data":"ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8"}
Nov 03 03:40:59.484298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.484230    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn" event={"ID":"a1baa566-ebed-4a3d-8232-c6cbd115d30e","Type":"ContainerStarted","Data":"9590d2b6ec035ec90f2cccb1bc385f959b9d23c668eb65b6dfb4eeb43230b6e7"}
Nov 03 03:40:59.484298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.484253    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:40:59.605988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.605940    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m" podStartSLOduration=1.605925512 podStartE2EDuration="1.605925512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:59.604333888 +0000 UTC m=+2350.835226307" watchObservedRunningTime="2026-11-03 03:40:59.605925512 +0000 UTC m=+2350.836817944"
Nov 03 03:40:59.733848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.733486    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn" podStartSLOduration=1.733464501 podStartE2EDuration="1.733464501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:40:59.729897863 +0000 UTC m=+2350.960790282" watchObservedRunningTime="2026-11-03 03:40:59.733464501 +0000 UTC m=+2350.964356926"
Nov 03 03:40:59.918076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:40:59.918046    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:01.125219 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:01.125179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:05.074385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:05.074351    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:05.241179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:05.241147    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:05.386213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:05.386180    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:10.244522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:10.244485    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:10.494666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:10.494586    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:41:10.494666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:10.494666    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:41:16.814348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:16.814318    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:21.043065 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:21.038933    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:41:21.096198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.096167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:21.270958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.270927    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:41:21.445993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.445958    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"]
Nov 03 03:41:21.446181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.446146    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t" podUID="00e9907a-a6e2-4abf-8311-40fdba1cf98e" containerName="payload-processing" containerID="cri-o://90de5a6a3e90bd6e4b0b893b2ea02bf2bff27206fce726ec98e10ed8af37d4d6" gracePeriod=30
Nov 03 03:41:21.507536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.507508    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"]
Nov 03 03:41:21.507767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.507746    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk" podUID="7f95be54-2b87-4ade-970e-0a93760a7da9" containerName="payload-pre-processing" containerID="cri-o://19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0" gracePeriod=30
Nov 03 03:41:21.604939 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.604870    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:21.634785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.634753    5515 generic.go:356] "Generic (PLEG): container finished" podID="00e9907a-a6e2-4abf-8311-40fdba1cf98e" containerID="90de5a6a3e90bd6e4b0b893b2ea02bf2bff27206fce726ec98e10ed8af37d4d6" exitCode=0
Nov 03 03:41:21.634909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.634817    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t" event={"ID":"00e9907a-a6e2-4abf-8311-40fdba1cf98e","Type":"ContainerDied","Data":"90de5a6a3e90bd6e4b0b893b2ea02bf2bff27206fce726ec98e10ed8af37d4d6"}
Nov 03 03:41:21.752933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.752906    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:21.776596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.776566    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:41:21.811432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.811411    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:41:21.828360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.828340    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume\") pod \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") "
Nov 03 03:41:21.828606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.828580    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76\" (UniqueName: \"kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76\") pod \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\" (UID: \"00e9907a-a6e2-4abf-8311-40fdba1cf98e\") "
Nov 03 03:41:21.828777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.828756    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume" pod "00e9907a-a6e2-4abf-8311-40fdba1cf98e" (UID: "00e9907a-a6e2-4abf-8311-40fdba1cf98e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:41:21.829030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.829006    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e9907a-a6e2-4abf-8311-40fdba1cf98e-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:21.830795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.830770    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76" pod "00e9907a-a6e2-4abf-8311-40fdba1cf98e" (UID: "00e9907a-a6e2-4abf-8311-40fdba1cf98e"). InnerVolumeSpecName "kube-api-access-v9g76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:41:21.929863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.929826    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume\") pod \"7f95be54-2b87-4ade-970e-0a93760a7da9\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") "
Nov 03 03:41:21.930034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.929974    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42\" (UniqueName: \"kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42\") pod \"7f95be54-2b87-4ade-970e-0a93760a7da9\" (UID: \"7f95be54-2b87-4ade-970e-0a93760a7da9\") "
Nov 03 03:41:21.930306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.930276    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume" pod "7f95be54-2b87-4ade-970e-0a93760a7da9" (UID: "7f95be54-2b87-4ade-970e-0a93760a7da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:41:21.930306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.930299    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9g76\" (UniqueName: \"kubernetes.io/projected/00e9907a-a6e2-4abf-8311-40fdba1cf98e-kube-api-access-v9g76\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:21.932193 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:21.932167    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42" pod "7f95be54-2b87-4ade-970e-0a93760a7da9" (UID: "7f95be54-2b87-4ade-970e-0a93760a7da9"). InnerVolumeSpecName "kube-api-access-gdn42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:41:22.032002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.031922    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdn42\" (UniqueName: \"kubernetes.io/projected/7f95be54-2b87-4ade-970e-0a93760a7da9-kube-api-access-gdn42\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:22.032002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.031956    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f95be54-2b87-4ade-970e-0a93760a7da9-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:22.607186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607154    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"]
Nov 03 03:41:22.607789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607774    5515 cpu_manager.go:414] "containerMap: removing container" podUID="00e9907a-a6e2-4abf-8311-40fdba1cf98e" containerName="payload-processing"
Nov 03 03:41:22.607845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607791    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e9907a-a6e2-4abf-8311-40fdba1cf98e" containerName="payload-processing"
Nov 03 03:41:22.607845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607816    5515 cpu_manager.go:414] "containerMap: removing container" podUID="7f95be54-2b87-4ade-970e-0a93760a7da9" containerName="payload-pre-processing"
Nov 03 03:41:22.607845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607821    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f95be54-2b87-4ade-970e-0a93760a7da9" containerName="payload-pre-processing"
Nov 03 03:41:22.607955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607916    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f95be54-2b87-4ade-970e-0a93760a7da9" containerName="payload-pre-processing"
Nov 03 03:41:22.607955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.607931    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e9907a-a6e2-4abf-8311-40fdba1cf98e" containerName="payload-processing"
Nov 03 03:41:22.611325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.611310    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.638665 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.638643    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.638789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.638683    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qj7\" (UniqueName: \"kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.638892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.638871    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.638945 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.638910    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.642124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.642099    5515 generic.go:356] "Generic (PLEG): container finished" podID="7f95be54-2b87-4ade-970e-0a93760a7da9" containerID="19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0" exitCode=0
Nov 03 03:41:22.642253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.642126    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk" event={"ID":"7f95be54-2b87-4ade-970e-0a93760a7da9","Type":"ContainerDied","Data":"19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0"}
Nov 03 03:41:22.642253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.642141    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"
Nov 03 03:41:22.642253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.642180    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk" event={"ID":"7f95be54-2b87-4ade-970e-0a93760a7da9","Type":"ContainerDied","Data":"46de732b46846f9068e6a8ca8603fa75280d98ecacc720bde14c060c02be8080"}
Nov 03 03:41:22.642253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.642212    5515 scope.go:122] "RemoveContainer" containerID="19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0"
Nov 03 03:41:22.643598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.643583    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"
Nov 03 03:41:22.643719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.643607    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t" event={"ID":"00e9907a-a6e2-4abf-8311-40fdba1cf98e","Type":"ContainerDied","Data":"a9e5552390b7b2b2345c9db7691f013783d2f27ccf80a58d35f04869861331bb"}
Nov 03 03:41:22.652644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.652621    5515 scope.go:122] "RemoveContainer" containerID="19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0"
Nov 03 03:41:22.652935 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:22.652912    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0\": container with ID starting with 19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0 not found: ID does not exist" containerID="19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0"
Nov 03 03:41:22.653021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.652945    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0"} err="failed to get container status \"19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0\": rpc error: code = NotFound desc = could not find container \"19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0\": container with ID starting with 19c78913c2f2146e15c21f83c8e170191860888e83c7ae56e8b835052c7b6fe0 not found: ID does not exist"
Nov 03 03:41:22.653021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.652970    5515 scope.go:122] "RemoveContainer" containerID="90de5a6a3e90bd6e4b0b893b2ea02bf2bff27206fce726ec98e10ed8af37d4d6"
Nov 03 03:41:22.732904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.732826    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-545809-serving-cert\""
Nov 03 03:41:22.735560 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.735524    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"]
Nov 03 03:41:22.740137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.740103    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qj7\" (UniqueName: \"kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.740284 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.740265    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.740332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.740299    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.740436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.740420    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.740522 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:22.740503    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-serving-cert: secret "maas-api-e2e-ipp-b-545809-serving-cert" not found
Nov 03 03:41:22.740620 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:22.740606    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls podName:d47cd588-cf6f-4d30-9400-b4246045d466 nodeName:}" failed. No retries permitted until 2026-11-03 03:41:23.240584492 +0000 UTC m=+2374.471476890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls") pod "maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" (UID: "d47cd588-cf6f-4d30-9400-b4246045d466") : secret "maas-api-e2e-ipp-b-545809-serving-cert" not found
Nov 03 03:41:22.741003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.740987    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.742815 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.742799    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.803190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.803158    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qj7\" (UniqueName: \"kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:22.830467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.830442    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:22.870303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.870278    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"]
Nov 03 03:41:22.876401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.876368    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8200d4-5d9c4d7f88-9km9t"]
Nov 03 03:41:22.927565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.927083    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e9907a-a6e2-4abf-8311-40fdba1cf98e" path="/var/lib/kubelet/pods/00e9907a-a6e2-4abf-8311-40fdba1cf98e/volumes"
Nov 03 03:41:22.989928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.989851    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"]
Nov 03 03:41:22.993586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.993558    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:22.994337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:22.994315    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8200d4-5956b46f47-2fgdk"]
Nov 03 03:41:23.140295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:23.140261    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:23.248184 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:23.248097    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:23.250752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:23.250730    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\") pod \"maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:23.525276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:23.525188    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:23.924010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:23.923983    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"]
Nov 03 03:41:23.926051 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:41:23.926026    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47cd588_cf6f_4d30_9400_b4246045d466.slice/crio-17d415c872daecf8a5e953e55f199fceded81a039e202375c8cc907968cd8332 WatchSource:0}: Error finding container 17d415c872daecf8a5e953e55f199fceded81a039e202375c8cc907968cd8332: Status 404 returned error can't find the container with id 17d415c872daecf8a5e953e55f199fceded81a039e202375c8cc907968cd8332
Nov 03 03:41:24.658638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:24.658600    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" event={"ID":"d47cd588-cf6f-4d30-9400-b4246045d466","Type":"ContainerStarted","Data":"17d415c872daecf8a5e953e55f199fceded81a039e202375c8cc907968cd8332"}
Nov 03 03:41:24.899728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:24.899698    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f95be54-2b87-4ade-970e-0a93760a7da9" path="/var/lib/kubelet/pods/7f95be54-2b87-4ade-970e-0a93760a7da9/volumes"
Nov 03 03:41:25.694426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:25.694395    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" event={"ID":"d47cd588-cf6f-4d30-9400-b4246045d466","Type":"ContainerStarted","Data":"6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6"}
Nov 03 03:41:25.694958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:25.694508    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:25.817771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:25.817718    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" podStartSLOduration=2.9549706589999998 podStartE2EDuration="3.817704366s" totalImagesPullingTime="862.733707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:25.817276203 +0000 UTC m=+2377.048168630" watchObservedRunningTime="2026-11-03 03:41:25.817704366 +0000 UTC m=+2377.048596795"
Nov 03 03:41:28.103857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:28.103826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:28.384573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:28.384086    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:28.531811 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:28.530863    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:28.789845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:28.789820    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:28.934496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:28.934464    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:30.696815 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:30.696777    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:31.193911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:31.193880    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:31.706207 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:31.706179    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:41:31.905391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:31.905352    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:32.442699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.442667    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:32.747800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.747690    5515 generic.go:356] "Generic (PLEG): container finished" podID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerID="f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896" exitCode=0
Nov 03 03:41:32.748308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.747916    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" event={"ID":"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c","Type":"ContainerDied","Data":"f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"}
Nov 03 03:41:32.749321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.748457    5515 scope.go:122] "RemoveContainer" containerID="f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"
Nov 03 03:41:32.750969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.750926    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:32.751099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.751025    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:32.751099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:32.751072    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:33.060279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.060179    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:41:33.513511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.513475    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"]
Nov 03 03:41:33.516660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.516643    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.591572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.580846    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b9c10a-dockercfg-2zsb4\""
Nov 03 03:41:33.591572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.581570    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b9c10a\""
Nov 03 03:41:33.591572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.591193    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"]
Nov 03 03:41:33.649756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.649721    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tbx\" (UniqueName: \"kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.649932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.649848    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.751578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.751522    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tbx\" (UniqueName: \"kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.752045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.751681    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.752588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.752534    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.756438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.756409    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" event={"ID":"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c","Type":"ContainerStarted","Data":"414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4"}
Nov 03 03:41:33.813384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.813310    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tbx\" (UniqueName: \"kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx\") pod \"payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.829377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.829344    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:33.851886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.851857    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:41:33.853390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.853359    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 03:41:33.853540 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.853421    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 03:41:33.888209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:33.888094    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:34.231392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.231363    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"]
Nov 03 03:41:34.232954 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:41:34.232927    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb146775_1c92_47fd_9f0a_1e29e00e82fd.slice/crio-2dd8024948748fcdde1b7c986f97641c3ef74b9b13f8584c6534f2c5a8bb313b WatchSource:0}: Error finding container 2dd8024948748fcdde1b7c986f97641c3ef74b9b13f8584c6534f2c5a8bb313b: Status 404 returned error can't find the container with id 2dd8024948748fcdde1b7c986f97641c3ef74b9b13f8584c6534f2c5a8bb313b
Nov 03 03:41:34.420998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.420963    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"]
Nov 03 03:41:34.423903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.423886    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.509828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.503790    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-29dd3f-openshift-default-dockercfg-6jcgx\""
Nov 03 03:41:34.516573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.511506    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:34.524575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.517700    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"]
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.570786    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjn8r\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.570856    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.570914    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.570973    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.571086    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.571121    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.571200    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.571258    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.576575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.571304    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697311    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697380    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697423    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697487    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjn8r\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697525    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697609    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.697653 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697656    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.698216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697742    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.698216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.697773    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.700294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.698632    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.700294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.698885    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.700294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.699195    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.700294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.699347    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.700294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.699469    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.703519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.703495    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.705728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.705703    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.724122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.724095    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"]
Nov 03 03:41:34.724339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.724317    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" containerID="cri-o://30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f" gracePeriod=30
Nov 03 03:41:34.787628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.786048    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.793064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.793033    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjn8r\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r\") pod \"e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") " pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:34.798635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.798498    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww" event={"ID":"db146775-1c92-47fd-9f0a-1e29e00e82fd","Type":"ContainerStarted","Data":"41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1"}
Nov 03 03:41:34.798635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.798573    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww" event={"ID":"db146775-1c92-47fd-9f0a-1e29e00e82fd","Type":"ContainerStarted","Data":"2dd8024948748fcdde1b7c986f97641c3ef74b9b13f8584c6534f2c5a8bb313b"}
Nov 03 03:41:34.798635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.798604    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:34.855654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.855617    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 03:41:34.855852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.855674    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 03:41:34.911715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:34.911652    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:35.007647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.007592    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww" podStartSLOduration=2.007578005 podStartE2EDuration="2.007578005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:35.005069824 +0000 UTC m=+2386.235962240" watchObservedRunningTime="2026-11-03 03:41:35.007578005 +0000 UTC m=+2386.238470422"
Nov 03 03:41:35.037261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.037229    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:35.337723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.337644    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:35.526590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.526528    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"]
Nov 03 03:41:35.529608 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:41:35.529567    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad47c06_3727_4e24_93f4_7137af0321ee.slice/crio-c17aebd1ec36baf52092184bc6755d9eab22160fc7b5eab5ffa4f74662719c10 WatchSource:0}: Error finding container c17aebd1ec36baf52092184bc6755d9eab22160fc7b5eab5ffa4f74662719c10: Status 404 returned error can't find the container with id c17aebd1ec36baf52092184bc6755d9eab22160fc7b5eab5ffa4f74662719c10
Nov 03 03:41:35.532241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.532205    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:35.532352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.532274    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:35.532352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.532301    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:35.806240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.806195    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" event={"ID":"4ad47c06-3727-4e24-93f4-7137af0321ee","Type":"ContainerStarted","Data":"3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"}
Nov 03 03:41:35.806699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.806261    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" event={"ID":"4ad47c06-3727-4e24-93f4-7137af0321ee","Type":"ContainerStarted","Data":"c17aebd1ec36baf52092184bc6755d9eab22160fc7b5eab5ffa4f74662719c10"}
Nov 03 03:41:35.852646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.852615    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 03:41:35.852827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.852669    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 03:41:35.933117 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:35.933016    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podStartSLOduration=1.932995638 podStartE2EDuration="1.932995638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:41:35.928761143 +0000 UTC m=+2387.159653561" watchObservedRunningTime="2026-11-03 03:41:35.932995638 +0000 UTC m=+2387.163888058"
Nov 03 03:41:36.039570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.039525    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:36.039900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.039877    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 03:41:36.040062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.040037    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 03:41:36.292631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.288984    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:41:36.292631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.289051    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:41:36.851906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.851871    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 03:41:36.852361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:36.851925    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 03:41:37.042296 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:37.042263    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:37.818645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:37.818610    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:37.819826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:37.819800    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:41:37.851774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:37.851743    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 03:41:37.851898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:37.851795    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 03:41:38.724725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:38.724685    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:38.852446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:38.852408    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 03:41:38.852655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:38.852460    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 03:41:38.945336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:38.945301    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:39.094212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:39.094133    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:39.857337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:39.857306    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:41:39.857739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:39.857716    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:41:39.858402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:39.858383    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:41:39.992387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:39.992336    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:41:40.080640 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080617    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.080793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080678    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.080793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080726    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.080793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080749    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.080793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080768    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.081015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080853    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.081015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080909    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:40.081015 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.080916    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.081166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081045    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.081166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081099    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data\") pod \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\" (UID: \"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67\") "
Nov 03 03:41:40.081166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081116    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:40.081327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081157    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:41:40.081388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081364    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:40.081448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081408    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:40.081682 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081663    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.081768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081687    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.081768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081702    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.081768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081714    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.081768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.081727    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.083749 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.083725    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:40.083852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.083747    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:41:40.083852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.083830    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "kube-api-access-tj7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:41:40.083964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.083850    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token" pod "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" (UID: "1709f66c-58e9-4bb3-a37f-6fe85ecc7b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:41:40.183265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.183173    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.183265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.183207    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.183265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.183222    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.183265 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.183237    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj7rp\" (UniqueName: \"kubernetes.io/projected/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67-kube-api-access-tj7rp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:40.536067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.536035    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:40.837210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.837125    5515 generic.go:356] "Generic (PLEG): container finished" podID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerID="30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f" exitCode=0
Nov 03 03:41:40.837210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.837170    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"
Nov 03 03:41:40.837423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.837205    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" event={"ID":"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67","Type":"ContainerDied","Data":"30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f"}
Nov 03 03:41:40.837423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.837259    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld" event={"ID":"1709f66c-58e9-4bb3-a37f-6fe85ecc7b67","Type":"ContainerDied","Data":"891a82c4a462bbd43272ccbe6928c7ebe7a57344b980e658683c21cefd0d841f"}
Nov 03 03:41:40.837423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.837281    5515 scope.go:122] "RemoveContainer" containerID="30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f"
Nov 03 03:41:40.856671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.856585    5515 scope.go:122] "RemoveContainer" containerID="30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f"
Nov 03 03:41:40.856926 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:40.856906    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f\": container with ID starting with 30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f not found: ID does not exist" containerID="30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f"
Nov 03 03:41:40.856967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:40.856935    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f"} err="failed to get container status \"30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f\": rpc error: code = NotFound desc = could not find container \"30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f\": container with ID starting with 30bc3dbe299c0242dba59da973d72c7e0f6efb692af82ecaeaae4e43ec4cc07f not found: ID does not exist"
Nov 03 03:41:41.005422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:41.005387    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"]
Nov 03 03:41:41.009488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:41.009463    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-9z6ld"]
Nov 03 03:41:41.977906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:41.977873    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:42.153348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:42.152172    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:42.895577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:42.895531    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" path="/var/lib/kubelet/pods/1709f66c-58e9-4bb3-a37f-6fe85ecc7b67/volumes"
Nov 03 03:41:45.767560 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:45.767515    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:45.823881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:45.823366    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:41:48.340490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:48.340447    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:41:50.053512 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:50.053484    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:41:50.056003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:50.055979    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:41:50.060000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:50.059984    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:41:50.087660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:50.087619    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:41:51.009882 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:51.009844    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:41:51.091950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:51.091918    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:52.921997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:52.921958    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"]
Nov 03 03:41:52.922495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:52.922201    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy" containerID="cri-o://4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13" gracePeriod=30
Nov 03 03:41:56.235744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:56.235706    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:58.195722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.195675    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:41:58.345195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345113    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345199    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345227    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345257    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345313    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345346    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345374    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345436    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345476    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:58.345717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345634    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket\") pod \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\" (UID: \"6d39bd5c-bd30-4447-9c98-0c96719fcdb2\") "
Nov 03 03:41:58.345892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345724    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:58.346079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.346016    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:58.346079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.345960    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:41:58.346252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.346184    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.346252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.346205    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.346252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.346220    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.346252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.346233    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.346447 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.346321    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:58.347826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.347799    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:41:58.347934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.347838    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "kube-api-access-r6rs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:41:58.348083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.348068    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:41:58.348145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.348104    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token" pod "6d39bd5c-bd30-4447-9c98-0c96719fcdb2" (UID: "6d39bd5c-bd30-4447-9c98-0c96719fcdb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:41:58.447153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.447075    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.447153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.447105    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.447153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.447116    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.447153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.447125    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6rs5\" (UniqueName: \"kubernetes.io/projected/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-kube-api-access-r6rs5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.447153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.447135    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d39bd5c-bd30-4447-9c98-0c96719fcdb2-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:41:58.951195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.951164    5515 generic.go:356] "Generic (PLEG): container finished" podID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerID="ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d" exitCode=0
Nov 03 03:41:58.951345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.951244    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" event={"ID":"5a3679b6-68f5-4844-9785-4e0f69ff9c79","Type":"ContainerDied","Data":"ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"}
Nov 03 03:41:58.951617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.951593    5515 scope.go:122] "RemoveContainer" containerID="ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"
Nov 03 03:41:58.952737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.952714    5515 generic.go:356] "Generic (PLEG): container finished" podID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerID="4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13" exitCode=0
Nov 03 03:41:58.952824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.952756    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" event={"ID":"6d39bd5c-bd30-4447-9c98-0c96719fcdb2","Type":"ContainerDied","Data":"4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13"}
Nov 03 03:41:58.952824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.952792    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k" event={"ID":"6d39bd5c-bd30-4447-9c98-0c96719fcdb2","Type":"ContainerDied","Data":"893340b1ff5f9d90ef5da11e2124a5d1178c3209a3fff8efcf3c7c74bcfdedbd"}
Nov 03 03:41:58.952824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.952818    5515 scope.go:122] "RemoveContainer" containerID="4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13"
Nov 03 03:41:58.952931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.952916    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"
Nov 03 03:41:58.953127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.953097    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:58.953196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.953179    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:58.953246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.953227    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:41:58.966529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.966512    5515 scope.go:122] "RemoveContainer" containerID="4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13"
Nov 03 03:41:58.966841 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:41:58.966823    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13\": container with ID starting with 4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13 not found: ID does not exist" containerID="4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13"
Nov 03 03:41:58.966899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:58.966849    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13"} err="failed to get container status \"4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13\": rpc error: code = NotFound desc = could not find container \"4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13\": container with ID starting with 4a1d8937f07f45a30d8a1ce5fdcdb11ff37fae229090b6e52d50a65893b9ad13 not found: ID does not exist"
Nov 03 03:41:59.241364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:59.241284    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"]
Nov 03 03:41:59.243483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:59.243462    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-f6b4k"]
Nov 03 03:41:59.390129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:59.390097    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:41:59.959846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:41:59.959809    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" event={"ID":"5a3679b6-68f5-4844-9785-4e0f69ff9c79","Type":"ContainerStarted","Data":"ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3"}
Nov 03 03:42:00.586588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:00.584810    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:42:00.587655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:00.587626    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 03:42:00.587822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:00.587689    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 03:42:00.895146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:00.895113    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" path="/var/lib/kubelet/pods/6d39bd5c-bd30-4447-9c98-0c96719fcdb2/volumes"
Nov 03 03:42:01.484008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:01.483976    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:01.586997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:01.586944    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 03:42:01.587411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:01.587007    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 03:42:02.584970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:02.584932    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 03:42:02.585251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:02.585222    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 03:42:02.689126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:02.689083    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:02.842603 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:02.842492    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:02.990242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:02.990205    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:03.586956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:03.586912    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 03:42:03.587169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:03.586978    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 03:42:04.194533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:04.194501    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:42:04.587012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:04.586907    5515 patch_prober.go:29] interesting pod/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 03:42:04.587173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:04.587009    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 03:42:05.589434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:05.589407    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:42:05.590473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:05.590451    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:42:07.710294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:07.710258    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:12.637978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:12.637946    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:17.629136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:17.629101    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:20.985116 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:42:20.985074    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:42:22.892609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:22.892522    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:24.098331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:24.098303    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:24.698896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:24.698868    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:24.849867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:24.849767    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:24.992905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:24.992832    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:28.536508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:28.536474    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:28.713162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:28.713126    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:28.905214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:28.905178    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:29.122920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:29.122880    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:29.269894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:29.269821    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:30.017499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:30.017467    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:31.758919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:31.758884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:31.930917 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:31.930854    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:35.379926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:35.379883    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:38.387909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:38.387878    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:39.673058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:39.673028    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:39.906000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:39.905963    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"]
Nov 03 03:42:40.010129 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:40.010049    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"]
Nov 03 03:42:40.265859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:40.265777    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:40.415612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:40.415581    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:42.416590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:42.416539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:44.906750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:44.906716    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq" podUID="4a0e5a39-b560-4e8e-a549-c84e1f6339a1" containerName="payload-processing" containerID="cri-o://4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0" gracePeriod=25
Nov 03 03:42:45.034450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.020617    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf" podUID="76deac35-77c1-4ab9-9ea8-8de8e48a5e22" containerName="payload-pre-processing" containerID="cri-o://777d1aca1faedd06140cee6f21026e688382890fa3bcfb7ac2e2439e02615b98" gracePeriod=25
Nov 03 03:42:45.133649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.133616    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:42:45.221359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.221330    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:42:45.248840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.248815    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume\") pod \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") "
Nov 03 03:42:45.249000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.248874    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d\" (UniqueName: \"kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d\") pod \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\" (UID: \"4a0e5a39-b560-4e8e-a549-c84e1f6339a1\") "
Nov 03 03:42:45.249253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.249211    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume" pod "4a0e5a39-b560-4e8e-a549-c84e1f6339a1" (UID: "4a0e5a39-b560-4e8e-a549-c84e1f6339a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:42:45.251281 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.251252    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d" pod "4a0e5a39-b560-4e8e-a549-c84e1f6339a1" (UID: "4a0e5a39-b560-4e8e-a549-c84e1f6339a1"). InnerVolumeSpecName "kube-api-access-hsv5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:42:45.259656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.259624    5515 generic.go:356] "Generic (PLEG): container finished" podID="76deac35-77c1-4ab9-9ea8-8de8e48a5e22" containerID="777d1aca1faedd06140cee6f21026e688382890fa3bcfb7ac2e2439e02615b98" exitCode=0
Nov 03 03:42:45.259778 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.259700    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf" event={"ID":"76deac35-77c1-4ab9-9ea8-8de8e48a5e22","Type":"ContainerDied","Data":"777d1aca1faedd06140cee6f21026e688382890fa3bcfb7ac2e2439e02615b98"}
Nov 03 03:42:45.261185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.261162    5515 generic.go:356] "Generic (PLEG): container finished" podID="4a0e5a39-b560-4e8e-a549-c84e1f6339a1" containerID="4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0" exitCode=0
Nov 03 03:42:45.261352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.261232    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"
Nov 03 03:42:45.261467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.261249    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq" event={"ID":"4a0e5a39-b560-4e8e-a549-c84e1f6339a1","Type":"ContainerDied","Data":"4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0"}
Nov 03 03:42:45.261539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.261475    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq" event={"ID":"4a0e5a39-b560-4e8e-a549-c84e1f6339a1","Type":"ContainerDied","Data":"54744bea9b2213c70b0b11e33a2cd1ed15b6e7763bf1b5627cbdc88ebb74ace4"}
Nov 03 03:42:45.261539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.261522    5515 scope.go:122] "RemoveContainer" containerID="4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0"
Nov 03 03:42:45.273895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.273875    5515 scope.go:122] "RemoveContainer" containerID="4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0"
Nov 03 03:42:45.274199 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:42:45.274176    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0\": container with ID starting with 4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0 not found: ID does not exist" containerID="4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0"
Nov 03 03:42:45.274267 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.274211    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0"} err="failed to get container status \"4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0\": rpc error: code = NotFound desc = could not find container \"4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0\": container with ID starting with 4a8e16edb1c8f19acf14d3cd28fd3d0b4ec47e9c55d27dc3c5ff3c50bd1bf6a0 not found: ID does not exist"
Nov 03 03:42:45.307639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.307620    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:42:45.350755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.350729    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume\") pod \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") "
Nov 03 03:42:45.350903 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.350885    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d\" (UniqueName: \"kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d\") pod \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\" (UID: \"76deac35-77c1-4ab9-9ea8-8de8e48a5e22\") "
Nov 03 03:42:45.351076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.351056    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume" pod "76deac35-77c1-4ab9-9ea8-8de8e48a5e22" (UID: "76deac35-77c1-4ab9-9ea8-8de8e48a5e22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:42:45.351394 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.351376    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:42:45.351441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.351403    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:42:45.351441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.351418    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsv5d\" (UniqueName: \"kubernetes.io/projected/4a0e5a39-b560-4e8e-a549-c84e1f6339a1-kube-api-access-hsv5d\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:42:45.353242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.353216    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d" pod "76deac35-77c1-4ab9-9ea8-8de8e48a5e22" (UID: "76deac35-77c1-4ab9-9ea8-8de8e48a5e22"). InnerVolumeSpecName "kube-api-access-jdc4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:42:45.452514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.452444    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdc4d\" (UniqueName: \"kubernetes.io/projected/76deac35-77c1-4ab9-9ea8-8de8e48a5e22-kube-api-access-jdc4d\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:42:45.582420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.582385    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"]
Nov 03 03:42:45.587126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.587097    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-545809-94c7c996f-w9vbq"]
Nov 03 03:42:45.625965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:45.625940    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:46.276820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.276738    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"
Nov 03 03:42:46.278824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.276761    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf" event={"ID":"76deac35-77c1-4ab9-9ea8-8de8e48a5e22","Type":"ContainerDied","Data":"5217c795c6fd66c862493f06da6d78abdeaa6d62772f305655d68cf6e9e4ded9"}
Nov 03 03:42:46.279000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.278852    5515 scope.go:122] "RemoveContainer" containerID="777d1aca1faedd06140cee6f21026e688382890fa3bcfb7ac2e2439e02615b98"
Nov 03 03:42:46.445895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.445859    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"]
Nov 03 03:42:46.496765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.496735    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-545809-7787fbd69f-pqftf"]
Nov 03 03:42:46.894877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.894848    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0e5a39-b560-4e8e-a549-c84e1f6339a1" path="/var/lib/kubelet/pods/4a0e5a39-b560-4e8e-a549-c84e1f6339a1/volumes"
Nov 03 03:42:46.895225 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:46.895212    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76deac35-77c1-4ab9-9ea8-8de8e48a5e22" path="/var/lib/kubelet/pods/76deac35-77c1-4ab9-9ea8-8de8e48a5e22/volumes"
Nov 03 03:42:47.479288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:47.479254    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:47.697478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:47.697432    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:47.848066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:47.847976    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:50.986166 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:42:50.986130    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:42:52.930320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:52.930290    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:42:57.834215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:42:57.834176    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:01.889151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:01.889114    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:02.724777 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:02.724748    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:06.990590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:06.989262    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:07.349191 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.349113    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"]
Nov 03 03:43:07.349340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.349302    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7" podUID="0cde483c-85f7-4693-85d3-41e47d0700c3" containerName="payload-processing" containerID="cri-o://c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753" gracePeriod=30
Nov 03 03:43:07.402123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.402091    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"]
Nov 03 03:43:07.402355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.402283    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk" podUID="bbe237be-7c05-47df-a59a-c20fb9f33a37" containerName="payload-pre-processing" containerID="cri-o://b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a" gracePeriod=30
Nov 03 03:43:07.511589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.511526    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:07.695190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.695136    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:07.747871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.747842    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:43:07.799196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.799170    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:43:07.851461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.851406    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume\") pod \"0cde483c-85f7-4693-85d3-41e47d0700c3\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") "
Nov 03 03:43:07.851702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.851493    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume\") pod \"bbe237be-7c05-47df-a59a-c20fb9f33a37\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") "
Nov 03 03:43:07.851702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.851600    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf\" (UniqueName: \"kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf\") pod \"0cde483c-85f7-4693-85d3-41e47d0700c3\" (UID: \"0cde483c-85f7-4693-85d3-41e47d0700c3\") "
Nov 03 03:43:07.851702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.851647    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf\" (UniqueName: \"kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf\") pod \"bbe237be-7c05-47df-a59a-c20fb9f33a37\" (UID: \"bbe237be-7c05-47df-a59a-c20fb9f33a37\") "
Nov 03 03:43:07.851905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.851854    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume" pod "0cde483c-85f7-4693-85d3-41e47d0700c3" (UID: "0cde483c-85f7-4693-85d3-41e47d0700c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:43:07.852017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.851904    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume" pod "bbe237be-7c05-47df-a59a-c20fb9f33a37" (UID: "bbe237be-7c05-47df-a59a-c20fb9f33a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:43:07.852261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.852236    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cde483c-85f7-4693-85d3-41e47d0700c3-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:07.852365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.852267    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbe237be-7c05-47df-a59a-c20fb9f33a37-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:07.854335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.854308    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf" pod "bbe237be-7c05-47df-a59a-c20fb9f33a37" (UID: "bbe237be-7c05-47df-a59a-c20fb9f33a37"). InnerVolumeSpecName "kube-api-access-sjxjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:43:07.854592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.854574    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf" pod "0cde483c-85f7-4693-85d3-41e47d0700c3" (UID: "0cde483c-85f7-4693-85d3-41e47d0700c3"). InnerVolumeSpecName "kube-api-access-qskjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:43:07.954656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.953710    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskjf\" (UniqueName: \"kubernetes.io/projected/0cde483c-85f7-4693-85d3-41e47d0700c3-kube-api-access-qskjf\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:07.954656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:07.953747    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjxjf\" (UniqueName: \"kubernetes.io/projected/bbe237be-7c05-47df-a59a-c20fb9f33a37-kube-api-access-sjxjf\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:08.416650 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.416613    5515 generic.go:356] "Generic (PLEG): container finished" podID="bbe237be-7c05-47df-a59a-c20fb9f33a37" containerID="b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a" exitCode=0
Nov 03 03:43:08.417097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.416658    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"
Nov 03 03:43:08.417097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.416706    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk" event={"ID":"bbe237be-7c05-47df-a59a-c20fb9f33a37","Type":"ContainerDied","Data":"b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a"}
Nov 03 03:43:08.417097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.416760    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk" event={"ID":"bbe237be-7c05-47df-a59a-c20fb9f33a37","Type":"ContainerDied","Data":"174348a99287d1b4e30b08672da98dc68961414caa577e49b3bcc3f7998afc18"}
Nov 03 03:43:08.417097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.416796    5515 scope.go:122] "RemoveContainer" containerID="b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a"
Nov 03 03:43:08.418272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.418247    5515 generic.go:356] "Generic (PLEG): container finished" podID="0cde483c-85f7-4693-85d3-41e47d0700c3" containerID="c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753" exitCode=0
Nov 03 03:43:08.418360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.418292    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"
Nov 03 03:43:08.418360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.418324    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7" event={"ID":"0cde483c-85f7-4693-85d3-41e47d0700c3","Type":"ContainerDied","Data":"c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753"}
Nov 03 03:43:08.418446 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.418380    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7" event={"ID":"0cde483c-85f7-4693-85d3-41e47d0700c3","Type":"ContainerDied","Data":"fc7a288d0caaae9bce484022611b660e5dd23dd1cbca21910dd0bb2f32221584"}
Nov 03 03:43:08.431335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.431309    5515 scope.go:122] "RemoveContainer" containerID="b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a"
Nov 03 03:43:08.431926 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:08.431903    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a\": container with ID starting with b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a not found: ID does not exist" containerID="b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a"
Nov 03 03:43:08.431998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.431937    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a"} err="failed to get container status \"b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a\": rpc error: code = NotFound desc = could not find container \"b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a\": container with ID starting with b964fae36f5fff0a918a3f1a1c616a491aa43ee4e88152c137875887d12b9c3a not found: ID does not exist"
Nov 03 03:43:08.431998 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.431956    5515 scope.go:122] "RemoveContainer" containerID="c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753"
Nov 03 03:43:08.440593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.440575    5515 scope.go:122] "RemoveContainer" containerID="c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753"
Nov 03 03:43:08.440859 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:08.440840    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753\": container with ID starting with c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753 not found: ID does not exist" containerID="c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753"
Nov 03 03:43:08.440920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.440868    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753"} err="failed to get container status \"c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753\": rpc error: code = NotFound desc = could not find container \"c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753\": container with ID starting with c2744be9266b998fa6e7b236180c316ee0b92b26d1427c6574cd93e4230af753 not found: ID does not exist"
Nov 03 03:43:08.601691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.601661    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"]
Nov 03 03:43:08.604685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.604662    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6e985-dc99b8b96-8cvh7"]
Nov 03 03:43:08.715738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.715655    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"]
Nov 03 03:43:08.719760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.719736    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6e985-978bcc759-55kwk"]
Nov 03 03:43:08.839609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.839578    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:08.925949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.925915    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cde483c-85f7-4693-85d3-41e47d0700c3" path="/var/lib/kubelet/pods/0cde483c-85f7-4693-85d3-41e47d0700c3/volumes"
Nov 03 03:43:08.926406 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:08.926390    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe237be-7c05-47df-a59a-c20fb9f33a37" path="/var/lib/kubelet/pods/bbe237be-7c05-47df-a59a-c20fb9f33a37/volumes"
Nov 03 03:43:09.198011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:09.197969    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:09.354937 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:09.354891    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:10.884573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:10.880155    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:11.412685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:11.412650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:14.192713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:14.192678    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:16.485516 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:16.485485    5515 generic.go:356] "Generic (PLEG): container finished" podID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerID="3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd" exitCode=0
Nov 03 03:43:16.486045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:16.485566    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" event={"ID":"4ad47c06-3727-4e24-93f4-7137af0321ee","Type":"ContainerDied","Data":"3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"}
Nov 03 03:43:16.486045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:16.485984    5515 scope.go:122] "RemoveContainer" containerID="3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"
Nov 03 03:43:16.487810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:16.487776    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:43:16.487932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:16.487863    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:43:16.487932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:16.487910    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:43:17.087791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:17.087753    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:17.495493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:17.495455    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" event={"ID":"4ad47c06-3727-4e24-93f4-7137af0321ee","Type":"ContainerStarted","Data":"861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b"}
Nov 03 03:43:17.972572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:17.972132    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:18.037667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:18.037633    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:43:18.038613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:18.038583    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 03:43:18.038740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:18.038642    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 03:43:19.038202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:19.038166    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 03:43:19.038636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:19.038223    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 03:43:19.161008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:19.160978    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:19.559445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:19.559400    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:19.774858 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:19.774826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:20.038565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:20.038519    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 03:43:20.038960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:20.038602    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 03:43:20.109740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:20.109705    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:21.010724 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:21.010682    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:43:21.038310 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:21.038274    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 03:43:21.038444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:21.038340    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 03:43:22.037705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:22.037669    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 03:43:22.038109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:22.037727    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 03:43:22.819328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:22.819292    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:43:23.042940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:23.042914    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:43:23.533034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:23.533007    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:43:24.093050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:24.093017    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:24.269750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:24.269721    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:24.432986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:24.432957    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:24.826880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:24.826805    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:27.370745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.370712    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:27.542509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.542477    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:27.790507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.790427    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"]
Nov 03 03:43:27.791126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791106    5515 cpu_manager.go:414] "containerMap: removing container" podUID="bbe237be-7c05-47df-a59a-c20fb9f33a37" containerName="payload-pre-processing"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791128    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe237be-7c05-47df-a59a-c20fb9f33a37" containerName="payload-pre-processing"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791145    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791152    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791165    5515 cpu_manager.go:414] "containerMap: removing container" podUID="76deac35-77c1-4ab9-9ea8-8de8e48a5e22" containerName="payload-pre-processing"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791174    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76deac35-77c1-4ab9-9ea8-8de8e48a5e22" containerName="payload-pre-processing"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791200    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4a0e5a39-b560-4e8e-a549-c84e1f6339a1" containerName="payload-processing"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791207    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0e5a39-b560-4e8e-a549-c84e1f6339a1" containerName="payload-processing"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791221    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy"
Nov 03 03:43:27.791242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791227    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791253    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0cde483c-85f7-4693-85d3-41e47d0700c3" containerName="payload-processing"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791260    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cde483c-85f7-4693-85d3-41e47d0700c3" containerName="payload-processing"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791404    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0e5a39-b560-4e8e-a549-c84e1f6339a1" containerName="payload-processing"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791418    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d39bd5c-bd30-4447-9c98-0c96719fcdb2" containerName="istio-proxy"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791433    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cde483c-85f7-4693-85d3-41e47d0700c3" containerName="payload-processing"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791447    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe237be-7c05-47df-a59a-c20fb9f33a37" containerName="payload-pre-processing"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791456    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1709f66c-58e9-4bb3-a37f-6fe85ecc7b67" containerName="istio-proxy"
Nov 03 03:43:27.791729 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.791464    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="76deac35-77c1-4ab9-9ea8-8de8e48a5e22" containerName="payload-pre-processing"
Nov 03 03:43:27.794042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.794022    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:27.849738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.849708    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a9e5cd\""
Nov 03 03:43:27.872905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.872846    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8lx\" (UniqueName: \"kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx\") pod \"payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:27.894037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.893951    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:27.906012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.905982    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a9e5cd-dockercfg-zns82\""
Nov 03 03:43:27.910536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.910504    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"]
Nov 03 03:43:27.916081 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.916047    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"]
Nov 03 03:43:27.948958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.948920    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:27.964867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.964840    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"]
Nov 03 03:43:27.996377 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.996342    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8lx\" (UniqueName: \"kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx\") pod \"payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:27.996569 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.996447    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:27.997182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:27.997162    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:28.059018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.058948    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8lx\" (UniqueName: \"kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx\") pod \"payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:28.097495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.097468    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6pn\" (UniqueName: \"kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn\") pod \"payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.097626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.097522    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.107014 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.106992    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:28.198465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.198429    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.198965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.198686    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6pn\" (UniqueName: \"kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn\") pod \"payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.199324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.199305    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.261099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.261069    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6pn\" (UniqueName: \"kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn\") pod \"payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.264912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.264888    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:28.480695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.480658    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"]
Nov 03 03:43:28.481365 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:43:28.481338    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ac2c61_e3c4_48f6_81c0_7caa4c894f91.slice/crio-4b22bc7aba541b8c8f8e3ce0a9c09e414979b0960b68cc5b96b716b7a113d428 WatchSource:0}: Error finding container 4b22bc7aba541b8c8f8e3ce0a9c09e414979b0960b68cc5b96b716b7a113d428: Status 404 returned error can't find the container with id 4b22bc7aba541b8c8f8e3ce0a9c09e414979b0960b68cc5b96b716b7a113d428
Nov 03 03:43:28.561757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.561726    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx" event={"ID":"69ac2c61-e3c4-48f6-81c0-7caa4c894f91","Type":"ContainerStarted","Data":"cb63305a29e404c4aa0a1be5fb584c70db0ad097332a0086b1699210701f41eb"}
Nov 03 03:43:28.561883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.561787    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx" event={"ID":"69ac2c61-e3c4-48f6-81c0-7caa4c894f91","Type":"ContainerStarted","Data":"4b22bc7aba541b8c8f8e3ce0a9c09e414979b0960b68cc5b96b716b7a113d428"}
Nov 03 03:43:28.562004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.561989    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:28.648745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.648720    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"]
Nov 03 03:43:28.650177 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:43:28.650153    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00342e8_823e_4555_a8a6_cd50bd08d790.slice/crio-2a3c3cd2607d5eae5645f45b0e04b932458beee06ce4e831112234eb46291ef0 WatchSource:0}: Error finding container 2a3c3cd2607d5eae5645f45b0e04b932458beee06ce4e831112234eb46291ef0: Status 404 returned error can't find the container with id 2a3c3cd2607d5eae5645f45b0e04b932458beee06ce4e831112234eb46291ef0
Nov 03 03:43:28.680471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:28.680419    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx" podStartSLOduration=1.680400692 podStartE2EDuration="1.680400692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:28.679712075 +0000 UTC m=+2499.910604493" watchObservedRunningTime="2026-11-03 03:43:28.680400692 +0000 UTC m=+2499.911293113"
Nov 03 03:43:29.568486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:29.568452    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj" event={"ID":"c00342e8-823e-4555-a8a6-cd50bd08d790","Type":"ContainerStarted","Data":"fffc0a65ab7863a7e2a4998a916cd3dee0da4954a1b4394f55942454e53547ae"}
Nov 03 03:43:29.568972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:29.568504    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj" event={"ID":"c00342e8-823e-4555-a8a6-cd50bd08d790","Type":"ContainerStarted","Data":"2a3c3cd2607d5eae5645f45b0e04b932458beee06ce4e831112234eb46291ef0"}
Nov 03 03:43:29.568972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:29.568526    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:29.687442 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:29.687387    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj" podStartSLOduration=2.687368383 podStartE2EDuration="2.687368383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:29.68597504 +0000 UTC m=+2500.916867471" watchObservedRunningTime="2026-11-03 03:43:29.687368383 +0000 UTC m=+2500.918260804"
Nov 03 03:43:31.982607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:31.982572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:32.181368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:32.181333    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:34.215391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:34.215358    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:43:36.908681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:36.908651    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"]
Nov 03 03:43:36.911629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:36.911612    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:36.964697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:36.964666    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3480a2-serving-cert\""
Nov 03 03:43:37.017885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.017852    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"]
Nov 03 03:43:37.105248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.105215    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.105428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.105268    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.105428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.105409    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.105582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.105487    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:37.105582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.105560    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99hh\" (UniqueName: \"kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.206717 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.206641    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99hh\" (UniqueName: \"kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.206884 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.206740    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.206884 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.206765    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.206884 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.206820    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.207007 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:37.206890    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-serving-cert: secret "maas-api-e2e-praxis-b-3480a2-serving-cert" not found
Nov 03 03:43:37.207007 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:37.206950    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls podName:8d45858c-016d-472c-9c8b-b887d2278878 nodeName:}" failed. No retries permitted until 2026-11-03 03:43:37.706935086 +0000 UTC m=+2508.937827482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls") pod "maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" (UID: "8d45858c-016d-472c-9c8b-b887d2278878") : secret "maas-api-e2e-praxis-b-3480a2-serving-cert" not found
Nov 03 03:43:37.207517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.207497    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.209430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.209411    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.267776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.267748    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:37.271300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.271273    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99hh\" (UniqueName: \"kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.523352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.523252    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:37.716302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.716268    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.718940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.718918    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:37.824743 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:37.824673    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:38.199083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:38.199059    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"]
Nov 03 03:43:38.200190 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:43:38.200164    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d45858c_016d_472c_9c8b_b887d2278878.slice/crio-b62813e99aed545b04253f11d58ba198e9ebd1914fd1cf9b52b33af419a80fe9 WatchSource:0}: Error finding container b62813e99aed545b04253f11d58ba198e9ebd1914fd1cf9b52b33af419a80fe9: Status 404 returned error can't find the container with id b62813e99aed545b04253f11d58ba198e9ebd1914fd1cf9b52b33af419a80fe9
Nov 03 03:43:38.624867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:38.624835    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" event={"ID":"8d45858c-016d-472c-9c8b-b887d2278878","Type":"ContainerStarted","Data":"b62813e99aed545b04253f11d58ba198e9ebd1914fd1cf9b52b33af419a80fe9"}
Nov 03 03:43:39.572130 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:39.572101    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:43:39.633979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:39.633950    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" event={"ID":"8d45858c-016d-472c-9c8b-b887d2278878","Type":"ContainerStarted","Data":"01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779"}
Nov 03 03:43:39.634140 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:39.634058    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:39.812481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:39.812431    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" podStartSLOduration=2.964962405 podStartE2EDuration="3.812414915s" totalImagesPullingTime="847.45251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:43:39.809670379 +0000 UTC m=+2511.040562794" watchObservedRunningTime="2026-11-03 03:43:39.812414915 +0000 UTC m=+2511.043307676"
Nov 03 03:43:40.577409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:40.577381    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:43:41.789184 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:41.789150    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"]
Nov 03 03:43:41.789637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:41.789332    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" podUID="d47cd588-cf6f-4d30-9400-b4246045d466" containerName="maas-api" containerID="cri-o://6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6" gracePeriod=30
Nov 03 03:43:41.905947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:41.905912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:42.084382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.084352    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:43:42.173695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.173670    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca\") pod \"d47cd588-cf6f-4d30-9400-b4246045d466\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") "
Nov 03 03:43:42.173838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.173726    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7\" (UniqueName: \"kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7\") pod \"d47cd588-cf6f-4d30-9400-b4246045d466\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") "
Nov 03 03:43:42.173838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.173781    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\") pod \"d47cd588-cf6f-4d30-9400-b4246045d466\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") "
Nov 03 03:43:42.173938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.173883    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs\") pod \"d47cd588-cf6f-4d30-9400-b4246045d466\" (UID: \"d47cd588-cf6f-4d30-9400-b4246045d466\") "
Nov 03 03:43:42.174051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.174023    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca" pod "d47cd588-cf6f-4d30-9400-b4246045d466" (UID: "d47cd588-cf6f-4d30-9400-b4246045d466"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:43:42.174466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.174426    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d47cd588-cf6f-4d30-9400-b4246045d466-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:42.176245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.176205    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs" pod "d47cd588-cf6f-4d30-9400-b4246045d466" (UID: "d47cd588-cf6f-4d30-9400-b4246045d466"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:43:42.176245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.176227    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls" pod "d47cd588-cf6f-4d30-9400-b4246045d466" (UID: "d47cd588-cf6f-4d30-9400-b4246045d466"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:43:42.176416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.176364    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7" pod "d47cd588-cf6f-4d30-9400-b4246045d466" (UID: "d47cd588-cf6f-4d30-9400-b4246045d466"). InnerVolumeSpecName "kube-api-access-z4qj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:43:42.275427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.275398    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:42.275427 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.275427    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47cd588-cf6f-4d30-9400-b4246045d466-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:42.275621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.275437    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qj7\" (UniqueName: \"kubernetes.io/projected/d47cd588-cf6f-4d30-9400-b4246045d466-kube-api-access-z4qj7\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:42.651996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.651966    5515 generic.go:356] "Generic (PLEG): container finished" podID="d47cd588-cf6f-4d30-9400-b4246045d466" containerID="6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6" exitCode=0
Nov 03 03:43:42.652268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.652025    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"
Nov 03 03:43:42.652268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.652050    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" event={"ID":"d47cd588-cf6f-4d30-9400-b4246045d466","Type":"ContainerDied","Data":"6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6"}
Nov 03 03:43:42.652268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.652102    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z" event={"ID":"d47cd588-cf6f-4d30-9400-b4246045d466","Type":"ContainerDied","Data":"17d415c872daecf8a5e953e55f199fceded81a039e202375c8cc907968cd8332"}
Nov 03 03:43:42.652268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.652123    5515 scope.go:122] "RemoveContainer" containerID="6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6"
Nov 03 03:43:42.662880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.662861    5515 scope.go:122] "RemoveContainer" containerID="6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6"
Nov 03 03:43:42.663172 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:42.663151    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6\": container with ID starting with 6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6 not found: ID does not exist" containerID="6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6"
Nov 03 03:43:42.663243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.663178    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6"} err="failed to get container status \"6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6\": rpc error: code = NotFound desc = could not find container \"6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6\": container with ID starting with 6d32019023affc675cb5daabb56b617cc030963dc098dd8334f88f3b357b6fc6 not found: ID does not exist"
Nov 03 03:43:42.851826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.851787    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"]
Nov 03 03:43:42.881620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.881521    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-545809-879f6c7dc-6z76z"]
Nov 03 03:43:42.898103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:42.898075    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47cd588-cf6f-4d30-9400-b4246045d466" path="/var/lib/kubelet/pods/d47cd588-cf6f-4d30-9400-b4246045d466/volumes"
Nov 03 03:43:44.027155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:44.027120    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:45.644907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:45.644881    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:43:48.544132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:48.544089    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:48.738391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:48.738361    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:50.139002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:50.138966    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:50.648718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:50.648680    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:50.660483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:50.660459    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"]
Nov 03 03:43:50.660727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:50.660704    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy" containerID="cri-o://ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3" gracePeriod=30
Nov 03 03:43:50.925472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:50.925433    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:51.002177 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:51.002139    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:43:52.246277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:52.246243    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:52.874571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:52.874342    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:53.208507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:53.208423    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:55.919688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:55.919664    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:43:56.044300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044221    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044324    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044366    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044424    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044455    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044535    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044608    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044640    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.044708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.044689    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert\") pod \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\" (UID: \"5a3679b6-68f5-4844-9785-4e0f69ff9c79\") "
Nov 03 03:43:56.045839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.045391    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:43:56.045839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.045465    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:43:56.046125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.046092    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:43:56.046391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.046363    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:43:56.046506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.046441    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:43:56.048211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.048114    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:43:56.048444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.048399    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:43:56.048527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.048466    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:43:56.048809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.048777    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5" pod "5a3679b6-68f5-4844-9785-4e0f69ff9c79" (UID: "5a3679b6-68f5-4844-9785-4e0f69ff9c79"). InnerVolumeSpecName "kube-api-access-brdf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:43:56.146355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146321    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146350    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146360    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdf5\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-kube-api-access-brdf5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146371    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146380    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146389    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146397    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146405    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.146638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.146414    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3679b6-68f5-4844-9785-4e0f69ff9c79-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:43:56.189736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.189709    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:56.740750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.740718    5515 generic.go:356] "Generic (PLEG): container finished" podID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerID="ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3" exitCode=0
Nov 03 03:43:56.740943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.740773    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" event={"ID":"5a3679b6-68f5-4844-9785-4e0f69ff9c79","Type":"ContainerDied","Data":"ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3"}
Nov 03 03:43:56.740943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.740777    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"
Nov 03 03:43:56.740943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.740814    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v" event={"ID":"5a3679b6-68f5-4844-9785-4e0f69ff9c79","Type":"ContainerDied","Data":"b433779d5878339117706532954ecb2ff987eedd60de5922c9299ed357e2e8e1"}
Nov 03 03:43:56.740943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.740835    5515 scope.go:122] "RemoveContainer" containerID="ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3"
Nov 03 03:43:56.751474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.751457    5515 scope.go:122] "RemoveContainer" containerID="ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"
Nov 03 03:43:56.760686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.760667    5515 scope.go:122] "RemoveContainer" containerID="ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3"
Nov 03 03:43:56.760953 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:56.760926    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3\": container with ID starting with ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3 not found: ID does not exist" containerID="ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3"
Nov 03 03:43:56.761059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.760956    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3"} err="failed to get container status \"ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3\": rpc error: code = NotFound desc = could not find container \"ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3\": container with ID starting with ce3b2716d82b44a53c7580c1e645a3f5cb8b678218b99f1f56ecbb4372d212f3 not found: ID does not exist"
Nov 03 03:43:56.761059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.760972    5515 scope.go:122] "RemoveContainer" containerID="ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"
Nov 03 03:43:56.761212 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:43:56.761196    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d\": container with ID starting with ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d not found: ID does not exist" containerID="ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"
Nov 03 03:43:56.761258 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.761216    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d"} err="failed to get container status \"ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d\": rpc error: code = NotFound desc = could not find container \"ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d\": container with ID starting with ae4876abc9735d883d235a58e84cc15a6be21daa7fe4fe7c1821231ae9cad93d not found: ID does not exist"
Nov 03 03:43:56.960200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:56.960170    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"]
Nov 03 03:43:57.011167 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:57.011080    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-545809-openshift-default-5d7857f8f5-4lz5v"]
Nov 03 03:43:57.301052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:57.300984    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:57.485132 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:57.485103    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:57.629742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:57.629718    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:43:58.894114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:43:58.894085    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" path="/var/lib/kubelet/pods/5a3679b6-68f5-4844-9785-4e0f69ff9c79/volumes"
Nov 03 03:44:00.550272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.550197    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:00.722374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.722248    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:00.968993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.968962    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"]
Nov 03 03:44:00.969538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969525    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy"
Nov 03 03:44:00.969628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969540    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy"
Nov 03 03:44:00.969628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969577    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d47cd588-cf6f-4d30-9400-b4246045d466" containerName="maas-api"
Nov 03 03:44:00.969628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969582    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47cd588-cf6f-4d30-9400-b4246045d466" containerName="maas-api"
Nov 03 03:44:00.969628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969603    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy"
Nov 03 03:44:00.969628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969608    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy"
Nov 03 03:44:00.969810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969695    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47cd588-cf6f-4d30-9400-b4246045d466" containerName="maas-api"
Nov 03 03:44:00.969810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969706    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy"
Nov 03 03:44:00.969810 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.969716    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3679b6-68f5-4844-9785-4e0f69ff9c79" containerName="istio-proxy"
Nov 03 03:44:00.974683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:00.974664    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.030807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.030782    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"]
Nov 03 03:44:01.078642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.078617    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2be7ec\""
Nov 03 03:44:01.126574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.117631    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"]
Nov 03 03:44:01.126574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.122624    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.137572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.132512    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjxsj\" (UniqueName: \"kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj\") pod \"payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.137572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.132596    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlltp\" (UniqueName: \"kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp\") pod \"payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.137572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.132652    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.137572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.132689    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.176622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.176590    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2be7ec-dockercfg-j4w9c\""
Nov 03 03:44:01.178934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.178904    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"]
Nov 03 03:44:01.234270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.234182    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjxsj\" (UniqueName: \"kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj\") pod \"payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.234270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.234231    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlltp\" (UniqueName: \"kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp\") pod \"payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.234500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.234276    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.234500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.234308    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.235248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.235223    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.235348 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.235229    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.284978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.284953    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:01.296705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.296680    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlltp\" (UniqueName: \"kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp\") pod \"payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.296816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.296771    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjxsj\" (UniqueName: \"kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj\") pod \"payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.450325 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.450289    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:01.588802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.588768    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:01.827357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.827325    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"]
Nov 03 03:44:01.827483 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:44:01.827388    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a9e5f0_b6aa_44fb_b643_05b2ea04a070.slice/crio-34e5c094dd9531891d40518ba8e6d7f5cbb0644d396f5c75af7e9cfd78fa6037 WatchSource:0}: Error finding container 34e5c094dd9531891d40518ba8e6d7f5cbb0644d396f5c75af7e9cfd78fa6037: Status 404 returned error can't find the container with id 34e5c094dd9531891d40518ba8e6d7f5cbb0644d396f5c75af7e9cfd78fa6037
Nov 03 03:44:01.964901 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:01.964872    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"]
Nov 03 03:44:01.966044 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:44:01.966022    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21bad23_57d9_483e_ac2d_dabcdc8f9f41.slice/crio-31db461b73b18e002bc426993cbe885415c4744d3583f6745a08f7d7cc0f3f3a WatchSource:0}: Error finding container 31db461b73b18e002bc426993cbe885415c4744d3583f6745a08f7d7cc0f3f3a: Status 404 returned error can't find the container with id 31db461b73b18e002bc426993cbe885415c4744d3583f6745a08f7d7cc0f3f3a
Nov 03 03:44:02.011108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.011079    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:02.183378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.183351    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:02.784395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.784362    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr" event={"ID":"e21bad23-57d9-483e-ac2d-dabcdc8f9f41","Type":"ContainerStarted","Data":"92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d"}
Nov 03 03:44:02.784886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.784409    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr" event={"ID":"e21bad23-57d9-483e-ac2d-dabcdc8f9f41","Type":"ContainerStarted","Data":"31db461b73b18e002bc426993cbe885415c4744d3583f6745a08f7d7cc0f3f3a"}
Nov 03 03:44:02.784886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.784463    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:02.786012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.785988    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk" event={"ID":"68a9e5f0-b6aa-44fb-b643-05b2ea04a070","Type":"ContainerStarted","Data":"90dc797f166100f0a181651d7176c7136741ce52d627499cc030d42bfb2439e9"}
Nov 03 03:44:02.786143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.786027    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk" event={"ID":"68a9e5f0-b6aa-44fb-b643-05b2ea04a070","Type":"ContainerStarted","Data":"34e5c094dd9531891d40518ba8e6d7f5cbb0644d396f5c75af7e9cfd78fa6037"}
Nov 03 03:44:02.786143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.786067    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:02.903683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:02.903635    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr" podStartSLOduration=2.903622318 podStartE2EDuration="2.903622318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:02.901426845 +0000 UTC m=+2534.132319274" watchObservedRunningTime="2026-11-03 03:44:02.903622318 +0000 UTC m=+2534.134514736"
Nov 03 03:44:03.035311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.034298    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk" podStartSLOduration=2.034278289 podStartE2EDuration="2.034278289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:03.030269623 +0000 UTC m=+2534.261162061" watchObservedRunningTime="2026-11-03 03:44:03.034278289 +0000 UTC m=+2534.265170708"
Nov 03 03:44:03.254910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.254875    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:03.776880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.776848    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"]
Nov 03 03:44:03.781043 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.781022    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.834711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.834672    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9b4cb2-openshift-default-dockercfg-p8vhq\""
Nov 03 03:44:03.842781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.842749    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"]
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.879758    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.879818    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.879909    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgvq\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.879935    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.879965    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.880100    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.880193    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.880227    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.882624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.880264    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.891590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.889499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:03.981057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981019    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981116    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgvq\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981146    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981168    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981251 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981247    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981309    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981344    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981381    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981422    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981514    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981741    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981849    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.981958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.981929    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.982204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.982182    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.983786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.983762    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:03.984108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:03.984084    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:04.047767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.047685    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgvq\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:04.053411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.051217    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:04.053411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.052941    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token\") pod \"e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") " pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:04.095773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.095747    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:04.527238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.527207    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"]
Nov 03 03:44:04.528678 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:44:04.528649    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509c0d87_f3b2_40e8_a21c_0ca99d6ae651.slice/crio-e39c1bc0e4b2a38699a1d67dd672e0cd21dcf8a769770295e5bbb80ffbb8b842 WatchSource:0}: Error finding container e39c1bc0e4b2a38699a1d67dd672e0cd21dcf8a769770295e5bbb80ffbb8b842: Status 404 returned error can't find the container with id e39c1bc0e4b2a38699a1d67dd672e0cd21dcf8a769770295e5bbb80ffbb8b842
Nov 03 03:44:04.531607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.531528    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:44:04.531691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.531650    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:44:04.531753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.531692    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:44:04.800241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.800159    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" event={"ID":"509c0d87-f3b2-40e8-a21c-0ca99d6ae651","Type":"ContainerStarted","Data":"42d8efa9f0ab2c9247a98fe6784ea9e7c4fcb00dc93700e08bab4a14ed48b43c"}
Nov 03 03:44:04.800241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.800212    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" event={"ID":"509c0d87-f3b2-40e8-a21c-0ca99d6ae651","Type":"ContainerStarted","Data":"e39c1bc0e4b2a38699a1d67dd672e0cd21dcf8a769770295e5bbb80ffbb8b842"}
Nov 03 03:44:04.924176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:04.924108    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podStartSLOduration=1.924088207 podStartE2EDuration="1.924088207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:44:04.922525319 +0000 UTC m=+2536.153417738" watchObservedRunningTime="2026-11-03 03:44:04.924088207 +0000 UTC m=+2536.154980626"
Nov 03 03:44:05.097071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:05.096982    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:05.103194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:05.103171    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:05.807746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:05.807716    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:05.808535 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:05.808515    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:44:06.549064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:06.549012    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:07.779660 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:07.779604    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:07.944607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:07.944572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:08.108613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:08.108449    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:11.096875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:11.096823    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:11.301127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:11.301091    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:12.023474 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:12.023427    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:13.795345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:13.795317    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:44:13.795868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:13.795390    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:44:17.191429 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:17.191365    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:17.335527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:17.335495    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:20.985146 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:44:20.985107    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:44:22.492825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:22.492779    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:27.735135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:27.735085    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:32.503437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:32.503376    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:32.658740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:32.658698    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:32.804711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:32.804629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:33.676644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:33.676612    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:44:37.723261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:37.723223    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:41.798221 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:41.798188    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj_abc45778-8f2e-4ff7-b200-5233d868dbe9/payload-processing/0.log"
Nov 03 03:44:43.863260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:43.863231    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:44.330940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:44.330913    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:48.911784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:48.911739    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:49.166894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:49.166818    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:50.983030 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:44:50.982992    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:44:54.207397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:54.207339    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:56.886378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:56.886323    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:44:59.128215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:44:59.128178    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:00.178433 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.178398    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"]
Nov 03 03:45:00.184478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.184458    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.201052 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.201028    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"]
Nov 03 03:45:00.204748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.204729    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"]
Nov 03 03:45:00.204848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.204794    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.208460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.208442    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.228416 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.228399    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"]
Nov 03 03:45:00.231867 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.231852    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"]
Nov 03 03:45:00.231961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.231946    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.235249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.235234    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.249487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.249470    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\""
Nov 03 03:45:00.285341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.285292    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"]
Nov 03 03:45:00.288392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.288365    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"]
Nov 03 03:45:00.325047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.325015    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"]
Nov 03 03:45:00.327342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.327316    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"]
Nov 03 03:45:00.357899 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.357866    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894625-xtjkb\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.358044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.357912    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.358044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358005    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.358119 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358085    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rtg\" (UniqueName: \"kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.358172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358158    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsldh\" (UniqueName: \"kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.358278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358265    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.358315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358295    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfmx\" (UniqueName: \"kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx\") pod \"maas-api-key-cleanup-29894625-xtjkb\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.358350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358316    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrts\" (UniqueName: \"kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts\") pod \"maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.358350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358334    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.358414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.358354    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxxm4\" (UniqueName: \"kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.390321 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.390290    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"]
Nov 03 03:45:00.459757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459667    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rtg\" (UniqueName: \"kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.459757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459738    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsldh\" (UniqueName: \"kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459806    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459836    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfmx\" (UniqueName: \"kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx\") pod \"maas-api-key-cleanup-29894625-xtjkb\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459854    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrts\" (UniqueName: \"kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts\") pod \"maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459878    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459914    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxxm4\" (UniqueName: \"kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459950    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894625-xtjkb\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.460025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.459991    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.460501 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.460053    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.460679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.460657    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894625-xtjkb\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.460773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.460717    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.460835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.460765    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.460835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.460818    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.460947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.460876    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.522131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.522101    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rtg\" (UniqueName: \"kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg\") pod \"maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.522838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.522813    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrts\" (UniqueName: \"kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts\") pod \"maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.522928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.522840    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxxm4\" (UniqueName: \"kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.522928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.522859    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsldh\" (UniqueName: \"kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh\") pod \"maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.522928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.522876    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfmx\" (UniqueName: \"kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx\") pod \"maas-api-key-cleanup-29894625-xtjkb\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.526562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.526521    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:00.545346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.545320    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:00.551607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.551586    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:00.798286 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.798215    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:00.819643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.819608    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:00.923672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.923643    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"]
Nov 03 03:45:00.929050 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:00.929019    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0b2bfd_c218_42dc_bf6f_ec53120614b5.slice/crio-b5f9cc53f448349eef378bd609c86661679cb16027c206cdd572c54fbb2ffa66 WatchSource:0}: Error finding container b5f9cc53f448349eef378bd609c86661679cb16027c206cdd572c54fbb2ffa66: Status 404 returned error can't find the container with id b5f9cc53f448349eef378bd609c86661679cb16027c206cdd572c54fbb2ffa66
Nov 03 03:45:00.972830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.972802    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"]
Nov 03 03:45:00.973596 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:00.973538    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc19370_fbe9_4490_bc0a_fb247bf11e81.slice/crio-18bb1baf32403d969e7a621833e4c575949aa90ffd62d824b6fb0ac4bbd8a64b WatchSource:0}: Error finding container 18bb1baf32403d969e7a621833e4c575949aa90ffd62d824b6fb0ac4bbd8a64b: Status 404 returned error can't find the container with id 18bb1baf32403d969e7a621833e4c575949aa90ffd62d824b6fb0ac4bbd8a64b
Nov 03 03:45:00.975397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:00.975366    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"]
Nov 03 03:45:00.975733 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:00.975713    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f5a921_9dcb_4f82_a761_4a867b2e60c3.slice/crio-590963ec33af3c37b52fbb874073c2327218a6a41e5a1815c4cefa22e91f5243 WatchSource:0}: Error finding container 590963ec33af3c37b52fbb874073c2327218a6a41e5a1815c4cefa22e91f5243: Status 404 returned error can't find the container with id 590963ec33af3c37b52fbb874073c2327218a6a41e5a1815c4cefa22e91f5243
Nov 03 03:45:01.046887 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:01.046851    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:01.137739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:01.137696    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" event={"ID":"efc19370-fbe9-4490-bc0a-fb247bf11e81","Type":"ContainerStarted","Data":"18bb1baf32403d969e7a621833e4c575949aa90ffd62d824b6fb0ac4bbd8a64b"}
Nov 03 03:45:01.139058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:01.139029    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerStarted","Data":"590963ec33af3c37b52fbb874073c2327218a6a41e5a1815c4cefa22e91f5243"}
Nov 03 03:45:01.140223 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:01.140194    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" event={"ID":"5b0b2bfd-c218-42dc-bf6f-ec53120614b5","Type":"ContainerStarted","Data":"b5f9cc53f448349eef378bd609c86661679cb16027c206cdd572c54fbb2ffa66"}
Nov 03 03:45:01.178489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:01.178461    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"]
Nov 03 03:45:01.178945 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:01.178920    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca541e02_95bb_4474_a942_dc8d9215ee47.slice/crio-ccca2b269f978426d1b04da755df19c4630ef5b197186d6e9cdd9d46b3dc0e1f WatchSource:0}: Error finding container ccca2b269f978426d1b04da755df19c4630ef5b197186d6e9cdd9d46b3dc0e1f: Status 404 returned error can't find the container with id ccca2b269f978426d1b04da755df19c4630ef5b197186d6e9cdd9d46b3dc0e1f
Nov 03 03:45:01.229294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:01.229263    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"]
Nov 03 03:45:01.230256 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:01.230232    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88a5f21_071a_4853_bdd9_3e8aff037b2e.slice/crio-5b407c681f13025dacb40f45622be5d76bd003e5dad4f901186721f3281f1251 WatchSource:0}: Error finding container 5b407c681f13025dacb40f45622be5d76bd003e5dad4f901186721f3281f1251: Status 404 returned error can't find the container with id 5b407c681f13025dacb40f45622be5d76bd003e5dad4f901186721f3281f1251
Nov 03 03:45:02.148605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.148571    5515 generic.go:356] "Generic (PLEG): container finished" podID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerID="4ebd95b0b43817a6adcf3f94082e21d5afc41519adfb11b4d01296aeeab14d18" exitCode=60
Nov 03 03:45:02.148793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.148668    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" event={"ID":"efc19370-fbe9-4490-bc0a-fb247bf11e81","Type":"ContainerDied","Data":"4ebd95b0b43817a6adcf3f94082e21d5afc41519adfb11b4d01296aeeab14d18"}
Nov 03 03:45:02.149031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.149012    5515 scope.go:122] "RemoveContainer" containerID="4ebd95b0b43817a6adcf3f94082e21d5afc41519adfb11b4d01296aeeab14d18"
Nov 03 03:45:02.150426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.150398    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerStarted","Data":"f96fbe451d891f68b04404c64c24082aaf296e01809c83481761e5f0acbd5d5e"}
Nov 03 03:45:02.150590 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.150451    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerStarted","Data":"ccca2b269f978426d1b04da755df19c4630ef5b197186d6e9cdd9d46b3dc0e1f"}
Nov 03 03:45:02.152090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.152065    5515 generic.go:356] "Generic (PLEG): container finished" podID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerID="9271e2b24dc3411f315f3fc62c04a10b68c1f9dc38e1656091dd1abc013c8c1f" exitCode=60
Nov 03 03:45:02.152216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.152146    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerDied","Data":"9271e2b24dc3411f315f3fc62c04a10b68c1f9dc38e1656091dd1abc013c8c1f"}
Nov 03 03:45:02.152434 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.152346    5515 scope.go:122] "RemoveContainer" containerID="9271e2b24dc3411f315f3fc62c04a10b68c1f9dc38e1656091dd1abc013c8c1f"
Nov 03 03:45:02.153793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.153778    5515 generic.go:356] "Generic (PLEG): container finished" podID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerID="b9a50b783a929726e3a30df8e209a9d9a61c14eb6b05d20a908bcdffcbadad73" exitCode=60
Nov 03 03:45:02.153870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.153825    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" event={"ID":"5b0b2bfd-c218-42dc-bf6f-ec53120614b5","Type":"ContainerDied","Data":"b9a50b783a929726e3a30df8e209a9d9a61c14eb6b05d20a908bcdffcbadad73"}
Nov 03 03:45:02.154037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.154024    5515 scope.go:122] "RemoveContainer" containerID="b9a50b783a929726e3a30df8e209a9d9a61c14eb6b05d20a908bcdffcbadad73"
Nov 03 03:45:02.155420 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.155402    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerStarted","Data":"f4f35e17782c68947eaf8bd79bf76ff8f03b926f5b3f8a5e175f0780aa3adb01"}
Nov 03 03:45:02.155512 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.155434    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerStarted","Data":"5b407c681f13025dacb40f45622be5d76bd003e5dad4f901186721f3281f1251"}
Nov 03 03:45:02.384425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.384378    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" podStartSLOduration=2.384362453 podStartE2EDuration="2.384362453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:02.383716865 +0000 UTC m=+2593.614609290" watchObservedRunningTime="2026-11-03 03:45:02.384362453 +0000 UTC m=+2593.615254870"
Nov 03 03:45:02.503131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.503079    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" podStartSLOduration=2.503062433 podStartE2EDuration="2.503062433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:02.500876035 +0000 UTC m=+2593.731768455" watchObservedRunningTime="2026-11-03 03:45:02.503062433 +0000 UTC m=+2593.733954896"
Nov 03 03:45:02.540468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:02.540439    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:03.164579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.164506    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerStarted","Data":"450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50"}
Nov 03 03:45:03.166539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.166513    5515 generic.go:356] "Generic (PLEG): container finished" podID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerID="a165ac55bb4a19262476c2306089a29cc4754e0b0c77cc3da834b5a8bb9f1673" exitCode=60
Nov 03 03:45:03.166676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.166589    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" event={"ID":"5b0b2bfd-c218-42dc-bf6f-ec53120614b5","Type":"ContainerDied","Data":"a165ac55bb4a19262476c2306089a29cc4754e0b0c77cc3da834b5a8bb9f1673"}
Nov 03 03:45:03.166676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.166647    5515 scope.go:122] "RemoveContainer" containerID="b9a50b783a929726e3a30df8e209a9d9a61c14eb6b05d20a908bcdffcbadad73"
Nov 03 03:45:03.166910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.166890    5515 scope.go:122] "RemoveContainer" containerID="a165ac55bb4a19262476c2306089a29cc4754e0b0c77cc3da834b5a8bb9f1673"
Nov 03 03:45:03.167224 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:03.167205    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j_odh-ai-gateway-infra(5b0b2bfd-c218-42dc-bf6f-ec53120614b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5"
Nov 03 03:45:03.168254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.168231    5515 generic.go:356] "Generic (PLEG): container finished" podID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerID="f4f35e17782c68947eaf8bd79bf76ff8f03b926f5b3f8a5e175f0780aa3adb01" exitCode=60
Nov 03 03:45:03.168362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.168339    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerDied","Data":"f4f35e17782c68947eaf8bd79bf76ff8f03b926f5b3f8a5e175f0780aa3adb01"}
Nov 03 03:45:03.168511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.168488    5515 scope.go:122] "RemoveContainer" containerID="f4f35e17782c68947eaf8bd79bf76ff8f03b926f5b3f8a5e175f0780aa3adb01"
Nov 03 03:45:03.170597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.170575    5515 generic.go:356] "Generic (PLEG): container finished" podID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerID="117f4e20b47af8fe56f49223bc34e6003d1321ffea9d02f0caede5909ae443cd" exitCode=60
Nov 03 03:45:03.170798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.170712    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" event={"ID":"efc19370-fbe9-4490-bc0a-fb247bf11e81","Type":"ContainerDied","Data":"117f4e20b47af8fe56f49223bc34e6003d1321ffea9d02f0caede5909ae443cd"}
Nov 03 03:45:03.170929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.170878    5515 scope.go:122] "RemoveContainer" containerID="117f4e20b47af8fe56f49223bc34e6003d1321ffea9d02f0caede5909ae443cd"
Nov 03 03:45:03.171741 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:03.171705    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894625-xtjkb_odh-ai-gateway-infra(efc19370-fbe9-4490-bc0a-fb247bf11e81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81"
Nov 03 03:45:03.172510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.172489    5515 generic.go:356] "Generic (PLEG): container finished" podID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerID="f96fbe451d891f68b04404c64c24082aaf296e01809c83481761e5f0acbd5d5e" exitCode=60
Nov 03 03:45:03.172676 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.172518    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerDied","Data":"f96fbe451d891f68b04404c64c24082aaf296e01809c83481761e5f0acbd5d5e"}
Nov 03 03:45:03.172813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.172797    5515 scope.go:122] "RemoveContainer" containerID="f96fbe451d891f68b04404c64c24082aaf296e01809c83481761e5f0acbd5d5e"
Nov 03 03:45:03.274771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.274746    5515 scope.go:122] "RemoveContainer" containerID="4ebd95b0b43817a6adcf3f94082e21d5afc41519adfb11b4d01296aeeab14d18"
Nov 03 03:45:03.284243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:03.284194    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" podStartSLOduration=3.284180757 podStartE2EDuration="3.284180757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:03.281998539 +0000 UTC m=+2594.512890958" watchObservedRunningTime="2026-11-03 03:45:03.284180757 +0000 UTC m=+2594.515073171"
Nov 03 03:45:04.182987 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.182950    5515 generic.go:356] "Generic (PLEG): container finished" podID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerID="450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50" exitCode=60
Nov 03 03:45:04.183370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.183001    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerDied","Data":"450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50"}
Nov 03 03:45:04.183370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.183081    5515 scope.go:122] "RemoveContainer" containerID="9271e2b24dc3411f315f3fc62c04a10b68c1f9dc38e1656091dd1abc013c8c1f"
Nov 03 03:45:04.183475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.183401    5515 scope.go:122] "RemoveContainer" containerID="450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50"
Nov 03 03:45:04.183767 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:04.183746    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf_odh-ai-gateway-infra(c7f5a921-9dcb-4f82-a761-4a867b2e60c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3"
Nov 03 03:45:04.190740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.190718    5515 scope.go:122] "RemoveContainer" containerID="a165ac55bb4a19262476c2306089a29cc4754e0b0c77cc3da834b5a8bb9f1673"
Nov 03 03:45:04.191126 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:04.191094    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j_odh-ai-gateway-infra(5b0b2bfd-c218-42dc-bf6f-ec53120614b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5"
Nov 03 03:45:04.192520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.192495    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerStarted","Data":"6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d"}
Nov 03 03:45:04.194517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.194497    5515 scope.go:122] "RemoveContainer" containerID="117f4e20b47af8fe56f49223bc34e6003d1321ffea9d02f0caede5909ae443cd"
Nov 03 03:45:04.194819 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:04.194799    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894625-xtjkb_odh-ai-gateway-infra(efc19370-fbe9-4490-bc0a-fb247bf11e81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81"
Nov 03 03:45:04.196074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.196053    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerStarted","Data":"123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"}
Nov 03 03:45:04.196296 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:04.196282    5515 scope.go:122] "RemoveContainer" containerID="123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"
Nov 03 03:45:04.196530 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:04.196512    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2_odh-ai-gateway-infra(ca541e02-95bb-4474-a942-dc8d9215ee47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47"
Nov 03 03:45:05.204059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.204001    5515 generic.go:356] "Generic (PLEG): container finished" podID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerID="123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409" exitCode=60
Nov 03 03:45:05.204517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.204087    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerDied","Data":"123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"}
Nov 03 03:45:05.204517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.204153    5515 scope.go:122] "RemoveContainer" containerID="f96fbe451d891f68b04404c64c24082aaf296e01809c83481761e5f0acbd5d5e"
Nov 03 03:45:05.204517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.204480    5515 scope.go:122] "RemoveContainer" containerID="123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"
Nov 03 03:45:05.204850 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:05.204821    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2_odh-ai-gateway-infra(ca541e02-95bb-4474-a942-dc8d9215ee47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47"
Nov 03 03:45:05.206148 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.206125    5515 scope.go:122] "RemoveContainer" containerID="450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50"
Nov 03 03:45:05.206449 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:05.206424    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf_odh-ai-gateway-infra(c7f5a921-9dcb-4f82-a761-4a867b2e60c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3"
Nov 03 03:45:05.207787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.207766    5515 generic.go:356] "Generic (PLEG): container finished" podID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerID="6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d" exitCode=60
Nov 03 03:45:05.207881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.207847    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerDied","Data":"6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d"}
Nov 03 03:45:05.208115 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.208099    5515 scope.go:122] "RemoveContainer" containerID="6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d"
Nov 03 03:45:05.208331 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:05.208315    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv_odh-ai-gateway-infra(d88a5f21-071a-4853-bdd9-3e8aff037b2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e"
Nov 03 03:45:05.216716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.216694    5515 scope.go:122] "RemoveContainer" containerID="f4f35e17782c68947eaf8bd79bf76ff8f03b926f5b3f8a5e175f0780aa3adb01"
Nov 03 03:45:05.896163 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:05.896136    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:06.044712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.044679    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:06.217527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.217435    5515 scope.go:122] "RemoveContainer" containerID="6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d"
Nov 03 03:45:06.217999 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:06.217830    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv_odh-ai-gateway-infra(d88a5f21-071a-4853-bdd9-3e8aff037b2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e"
Nov 03 03:45:06.219355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.219334    5515 scope.go:122] "RemoveContainer" containerID="123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"
Nov 03 03:45:06.219661 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:06.219635    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2_odh-ai-gateway-infra(ca541e02-95bb-4474-a942-dc8d9215ee47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47"
Nov 03 03:45:06.356278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.356246    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"]
Nov 03 03:45:06.467356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.467322    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"]
Nov 03 03:45:06.722010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.721964    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:06.868918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:06.868886    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:07.978819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:07.978766    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:45:08.191282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:08.191222    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:08.342710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:08.342614    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:10.699734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.699702    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"]
Nov 03 03:45:10.703728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.703710    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.759163 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.759133    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-2be7ec-serving-cert\""
Nov 03 03:45:10.813029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.812996    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"]
Nov 03 03:45:10.824659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.824624    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.824812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.824697    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.824876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.824804    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.824876 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.824838    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gss4t\" (UniqueName: \"kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.895738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.895701    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:10.926084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.926044    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.926231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.926095    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.926231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.926141    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.926231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.926156    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gss4t\" (UniqueName: \"kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.926371 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:10.926291    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-serving-cert: secret "maas-api-e2e-praxis-a-2be7ec-serving-cert" not found
Nov 03 03:45:10.926371 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:10.926367    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls podName:f5db33db-363e-4986-ac5c-97875562eb05 nodeName:}" failed. No retries permitted until 2026-11-03 03:45:11.42634674 +0000 UTC m=+2602.657239137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls") pod "maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" (UID: "f5db33db-363e-4986-ac5c-97875562eb05") : secret "maas-api-e2e-praxis-a-2be7ec-serving-cert" not found
Nov 03 03:45:10.926770 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.926753    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.928859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.928835    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:10.990579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:10.990471    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gss4t\" (UniqueName: \"kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:11.119921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.119883    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:11.265735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.265658    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:11.356575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.356498    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs" podUID="734b45c9-9083-4105-890a-3a9f18a6494f" containerName="payload-processing" containerID="cri-o://628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50" gracePeriod=25
Nov 03 03:45:11.436780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.436752    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:11.439110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.439091    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:11.468156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.468123    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv" podUID="0c0d3818-5409-4e51-ab2f-296ae66a954c" containerName="payload-pre-processing" containerID="cri-o://952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b" gracePeriod=25
Nov 03 03:45:11.617102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.617080    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:11.625302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.625276    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:45:11.742139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.741049    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm\" (UniqueName: \"kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm\") pod \"734b45c9-9083-4105-890a-3a9f18a6494f\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") "
Nov 03 03:45:11.742139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.741171    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume\") pod \"734b45c9-9083-4105-890a-3a9f18a6494f\" (UID: \"734b45c9-9083-4105-890a-3a9f18a6494f\") "
Nov 03 03:45:11.747323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.743953    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume" pod "734b45c9-9083-4105-890a-3a9f18a6494f" (UID: "734b45c9-9083-4105-890a-3a9f18a6494f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:11.747323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.746905    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm" pod "734b45c9-9083-4105-890a-3a9f18a6494f" (UID: "734b45c9-9083-4105-890a-3a9f18a6494f"). InnerVolumeSpecName "kube-api-access-6qxwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:11.751586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.751532    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:45:11.844980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.844917    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume\") pod \"0c0d3818-5409-4e51-ab2f-296ae66a954c\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") "
Nov 03 03:45:11.844980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.844966    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq\" (UniqueName: \"kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq\") pod \"0c0d3818-5409-4e51-ab2f-296ae66a954c\" (UID: \"0c0d3818-5409-4e51-ab2f-296ae66a954c\") "
Nov 03 03:45:11.845261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.845240    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume" pod "0c0d3818-5409-4e51-ab2f-296ae66a954c" (UID: "0c0d3818-5409-4e51-ab2f-296ae66a954c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:11.845431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.845411    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qxwm\" (UniqueName: \"kubernetes.io/projected/734b45c9-9083-4105-890a-3a9f18a6494f-kube-api-access-6qxwm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:11.845431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.845429    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d3818-5409-4e51-ab2f-296ae66a954c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:11.845580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.845439    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734b45c9-9083-4105-890a-3a9f18a6494f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:11.847628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.847600    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq" pod "0c0d3818-5409-4e51-ab2f-296ae66a954c" (UID: "0c0d3818-5409-4e51-ab2f-296ae66a954c"). InnerVolumeSpecName "kube-api-access-9nmbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:11.912462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.912438    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:11.947101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:11.947074    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmbq\" (UniqueName: \"kubernetes.io/projected/0c0d3818-5409-4e51-ab2f-296ae66a954c-kube-api-access-9nmbq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:12.059066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.059047    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"]
Nov 03 03:45:12.059969 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:12.059945    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5db33db_363e_4986_ac5c_97875562eb05.slice/crio-029395295fef6b4c7b5a527c87b8f5ec855654bde2363d709e7003ab445a03b3 WatchSource:0}: Error finding container 029395295fef6b4c7b5a527c87b8f5ec855654bde2363d709e7003ab445a03b3: Status 404 returned error can't find the container with id 029395295fef6b4c7b5a527c87b8f5ec855654bde2363d709e7003ab445a03b3
Nov 03 03:45:12.061756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.061736    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:45:12.272227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.272195    5515 generic.go:356] "Generic (PLEG): container finished" podID="0c0d3818-5409-4e51-ab2f-296ae66a954c" containerID="952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b" exitCode=0
Nov 03 03:45:12.272425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.272241    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"
Nov 03 03:45:12.272425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.272283    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv" event={"ID":"0c0d3818-5409-4e51-ab2f-296ae66a954c","Type":"ContainerDied","Data":"952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b"}
Nov 03 03:45:12.272425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.272343    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv" event={"ID":"0c0d3818-5409-4e51-ab2f-296ae66a954c","Type":"ContainerDied","Data":"2dd392f69c63f3ea5d74fb9b72bfff11527b16f36640fb231b083155f4cf4196"}
Nov 03 03:45:12.272425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.272370    5515 scope.go:122] "RemoveContainer" containerID="952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b"
Nov 03 03:45:12.273755 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.273734    5515 generic.go:356] "Generic (PLEG): container finished" podID="734b45c9-9083-4105-890a-3a9f18a6494f" containerID="628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50" exitCode=0
Nov 03 03:45:12.273880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.273785    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"
Nov 03 03:45:12.273880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.273829    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs" event={"ID":"734b45c9-9083-4105-890a-3a9f18a6494f","Type":"ContainerDied","Data":"628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50"}
Nov 03 03:45:12.274020 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.273959    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs" event={"ID":"734b45c9-9083-4105-890a-3a9f18a6494f","Type":"ContainerDied","Data":"cb70d34b7a49fb7036f5b55bf96d414ea947e7243afc697d2122d4dcad7e0627"}
Nov 03 03:45:12.275760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.275733    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" event={"ID":"f5db33db-363e-4986-ac5c-97875562eb05","Type":"ContainerStarted","Data":"029395295fef6b4c7b5a527c87b8f5ec855654bde2363d709e7003ab445a03b3"}
Nov 03 03:45:12.282263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.282247    5515 scope.go:122] "RemoveContainer" containerID="952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b"
Nov 03 03:45:12.282594 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:12.282537    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b\": container with ID starting with 952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b not found: ID does not exist" containerID="952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b"
Nov 03 03:45:12.282671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.282604    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b"} err="failed to get container status \"952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b\": rpc error: code = NotFound desc = could not find container \"952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b\": container with ID starting with 952edbe56ea041ef6953a3a5e320668e58196c1072e7a08735bdee9422fcb93b not found: ID does not exist"
Nov 03 03:45:12.282671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.282624    5515 scope.go:122] "RemoveContainer" containerID="628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50"
Nov 03 03:45:12.291768 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.291750    5515 scope.go:122] "RemoveContainer" containerID="628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50"
Nov 03 03:45:12.292017 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:12.292000    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50\": container with ID starting with 628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50 not found: ID does not exist" containerID="628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50"
Nov 03 03:45:12.292090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.292025    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50"} err="failed to get container status \"628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50\": rpc error: code = NotFound desc = could not find container \"628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50\": container with ID starting with 628ccb973725cec5914c4939d36a8a3ba891691ff0b2d1d9bf4a3bc37db09d50 not found: ID does not exist"
Nov 03 03:45:12.443353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.443205    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"]
Nov 03 03:45:12.456056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.456013    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-754aeb-74b8bc979-j7svv"]
Nov 03 03:45:12.565572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.565479    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"]
Nov 03 03:45:12.570155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.570129    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-754aeb-69587fdc7f-x6bgs"]
Nov 03 03:45:12.894417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.894387    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0d3818-5409-4e51-ab2f-296ae66a954c" path="/var/lib/kubelet/pods/0c0d3818-5409-4e51-ab2f-296ae66a954c/volumes"
Nov 03 03:45:12.894826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:12.894799    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734b45c9-9083-4105-890a-3a9f18a6494f" path="/var/lib/kubelet/pods/734b45c9-9083-4105-890a-3a9f18a6494f/volumes"
Nov 03 03:45:13.285414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:13.285320    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" event={"ID":"f5db33db-363e-4986-ac5c-97875562eb05","Type":"ContainerStarted","Data":"467d7216184f0cd5a33543e7394a433b1671637d6fbd9a1838894eea1cf25687"}
Nov 03 03:45:13.285628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:13.285568    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:13.406817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:13.406769    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" podStartSLOduration=2.492173567 podStartE2EDuration="3.406753603s" totalImagesPullingTime="914.580036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:13.404796893 +0000 UTC m=+2604.635689311" watchObservedRunningTime="2026-11-03 03:45:13.406753603 +0000 UTC m=+2604.637646020"
Nov 03 03:45:13.889385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:13.889346    5515 scope.go:122] "RemoveContainer" containerID="117f4e20b47af8fe56f49223bc34e6003d1321ffea9d02f0caede5909ae443cd"
Nov 03 03:45:14.296567 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:14.296506    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" event={"ID":"efc19370-fbe9-4490-bc0a-fb247bf11e81","Type":"ContainerStarted","Data":"bc2490cbdf4ab43601b85e4746eda5e95590f3644ce1bd1081880ab5faef1d88"}
Nov 03 03:45:14.414235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:14.414191    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" podStartSLOduration=14.414177694 podStartE2EDuration="14.414177694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:14.413460975 +0000 UTC m=+2605.644353392" watchObservedRunningTime="2026-11-03 03:45:14.414177694 +0000 UTC m=+2605.645070111"
Nov 03 03:45:14.886940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:14.886908    5515 scope.go:122] "RemoveContainer" containerID="123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"
Nov 03 03:45:14.887089 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:14.887021    5515 scope.go:122] "RemoveContainer" containerID="a165ac55bb4a19262476c2306089a29cc4754e0b0c77cc3da834b5a8bb9f1673"
Nov 03 03:45:15.306843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.306816    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" event={"ID":"5b0b2bfd-c218-42dc-bf6f-ec53120614b5","Type":"ContainerStarted","Data":"5923bcadc960a6b5b3390cf20ce5d99e0a0ff075d9970aa50e6e5b258309ea29"}
Nov 03 03:45:15.308656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.308632    5515 generic.go:356] "Generic (PLEG): container finished" podID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerID="bc2490cbdf4ab43601b85e4746eda5e95590f3644ce1bd1081880ab5faef1d88" exitCode=60
Nov 03 03:45:15.308791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.308707    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" event={"ID":"efc19370-fbe9-4490-bc0a-fb247bf11e81","Type":"ContainerDied","Data":"bc2490cbdf4ab43601b85e4746eda5e95590f3644ce1bd1081880ab5faef1d88"}
Nov 03 03:45:15.308791 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.308780    5515 scope.go:122] "RemoveContainer" containerID="117f4e20b47af8fe56f49223bc34e6003d1321ffea9d02f0caede5909ae443cd"
Nov 03 03:45:15.309069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.309049    5515 scope.go:122] "RemoveContainer" containerID="bc2490cbdf4ab43601b85e4746eda5e95590f3644ce1bd1081880ab5faef1d88"
Nov 03 03:45:15.309330 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:15.309312    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894625-xtjkb_odh-ai-gateway-infra(efc19370-fbe9-4490-bc0a-fb247bf11e81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81"
Nov 03 03:45:15.310655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.310635    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerStarted","Data":"857ed33220300ee688b3f531de21def32b1eb8e6d1e75889d053fa55f9c13d84"}
Nov 03 03:45:15.425859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.425824    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"]
Nov 03 03:45:15.446631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.446581    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" podStartSLOduration=15.44656732 podStartE2EDuration="15.44656732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:15.444969634 +0000 UTC m=+2606.675862051" watchObservedRunningTime="2026-11-03 03:45:15.44656732 +0000 UTC m=+2606.677459732"
Nov 03 03:45:15.886456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:15.886436    5515 scope.go:122] "RemoveContainer" containerID="450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50"
Nov 03 03:45:16.040366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.040293    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:16.318914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.318846    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerStarted","Data":"2203e6c0feac02983b94453b3c23e90d74212ae985e127cd44b71483c8b8d0e1"}
Nov 03 03:45:16.320764 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.320741    5515 generic.go:356] "Generic (PLEG): container finished" podID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerID="5923bcadc960a6b5b3390cf20ce5d99e0a0ff075d9970aa50e6e5b258309ea29" exitCode=60
Nov 03 03:45:16.320882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.320813    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" event={"ID":"5b0b2bfd-c218-42dc-bf6f-ec53120614b5","Type":"ContainerDied","Data":"5923bcadc960a6b5b3390cf20ce5d99e0a0ff075d9970aa50e6e5b258309ea29"}
Nov 03 03:45:16.320882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.320857    5515 scope.go:122] "RemoveContainer" containerID="a165ac55bb4a19262476c2306089a29cc4754e0b0c77cc3da834b5a8bb9f1673"
Nov 03 03:45:16.321111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.321096    5515 scope.go:122] "RemoveContainer" containerID="5923bcadc960a6b5b3390cf20ce5d99e0a0ff075d9970aa50e6e5b258309ea29"
Nov 03 03:45:16.321392 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:16.321367    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j_odh-ai-gateway-infra(5b0b2bfd-c218-42dc-bf6f-ec53120614b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5"
Nov 03 03:45:16.324663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.324632    5515 generic.go:356] "Generic (PLEG): container finished" podID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerID="857ed33220300ee688b3f531de21def32b1eb8e6d1e75889d053fa55f9c13d84" exitCode=60
Nov 03 03:45:16.324800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.324688    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerDied","Data":"857ed33220300ee688b3f531de21def32b1eb8e6d1e75889d053fa55f9c13d84"}
Nov 03 03:45:16.324968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.324952    5515 scope.go:122] "RemoveContainer" containerID="857ed33220300ee688b3f531de21def32b1eb8e6d1e75889d053fa55f9c13d84"
Nov 03 03:45:16.325169 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:16.325153    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2_odh-ai-gateway-infra(ca541e02-95bb-4474-a942-dc8d9215ee47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47"
Nov 03 03:45:16.334905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.334888    5515 scope.go:122] "RemoveContainer" containerID="123fcc70fa8cb88ebd66fbca2f5a8748d15ef73de21a45bc741b768fd90ac409"
Nov 03 03:45:16.438696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.438665    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"]
Nov 03 03:45:16.478018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.477995    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:16.607409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.607387    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx\" (UniqueName: \"kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx\") pod \"efc19370-fbe9-4490-bc0a-fb247bf11e81\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") "
Nov 03 03:45:16.607591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.607466    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca\") pod \"efc19370-fbe9-4490-bc0a-fb247bf11e81\" (UID: \"efc19370-fbe9-4490-bc0a-fb247bf11e81\") "
Nov 03 03:45:16.607837 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.607813    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca" pod "efc19370-fbe9-4490-bc0a-fb247bf11e81" (UID: "efc19370-fbe9-4490-bc0a-fb247bf11e81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:16.607941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.607923    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc19370-fbe9-4490-bc0a-fb247bf11e81-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:16.609699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.609680    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx" pod "efc19370-fbe9-4490-bc0a-fb247bf11e81" (UID: "efc19370-fbe9-4490-bc0a-fb247bf11e81"). InnerVolumeSpecName "kube-api-access-rlfmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:16.691445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.691413    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"]
Nov 03 03:45:16.708647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:16.708614    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfmx\" (UniqueName: \"kubernetes.io/projected/efc19370-fbe9-4490-bc0a-fb247bf11e81-kube-api-access-rlfmx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:17.332537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.332454    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"
Nov 03 03:45:17.332537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.332480    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb" event={"ID":"efc19370-fbe9-4490-bc0a-fb247bf11e81","Type":"ContainerDied","Data":"18bb1baf32403d969e7a621833e4c575949aa90ffd62d824b6fb0ac4bbd8a64b"}
Nov 03 03:45:17.333106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.332559    5515 scope.go:122] "RemoveContainer" containerID="bc2490cbdf4ab43601b85e4746eda5e95590f3644ce1bd1081880ab5faef1d88"
Nov 03 03:45:17.337460 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.337436    5515 generic.go:356] "Generic (PLEG): container finished" podID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerID="2203e6c0feac02983b94453b3c23e90d74212ae985e127cd44b71483c8b8d0e1" exitCode=60
Nov 03 03:45:17.337600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.337502    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerDied","Data":"2203e6c0feac02983b94453b3c23e90d74212ae985e127cd44b71483c8b8d0e1"}
Nov 03 03:45:17.338064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.338047    5515 scope.go:122] "RemoveContainer" containerID="2203e6c0feac02983b94453b3c23e90d74212ae985e127cd44b71483c8b8d0e1"
Nov 03 03:45:17.338328 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:17.338306    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf_odh-ai-gateway-infra(c7f5a921-9dcb-4f82-a761-4a867b2e60c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3"
Nov 03 03:45:17.345413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.345395    5515 scope.go:122] "RemoveContainer" containerID="450c4e4ba4b5f1549b173e3ce96235792b2d3ad1ecad1070870f818edf2e9f50"
Nov 03 03:45:17.447718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.447687    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"]
Nov 03 03:45:17.501490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.501464    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:17.535828 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.535809    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:17.619137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.619106    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca\") pod \"ca541e02-95bb-4474-a942-dc8d9215ee47\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") "
Nov 03 03:45:17.619303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.619174    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg\" (UniqueName: \"kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg\") pod \"ca541e02-95bb-4474-a942-dc8d9215ee47\" (UID: \"ca541e02-95bb-4474-a942-dc8d9215ee47\") "
Nov 03 03:45:17.619422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.619403    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca\") pod \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") "
Nov 03 03:45:17.619480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.619450    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca" pod "ca541e02-95bb-4474-a942-dc8d9215ee47" (UID: "ca541e02-95bb-4474-a942-dc8d9215ee47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:17.619782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.619762    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca" pod "5b0b2bfd-c218-42dc-bf6f-ec53120614b5" (UID: "5b0b2bfd-c218-42dc-bf6f-ec53120614b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:17.620084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.620065    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca541e02-95bb-4474-a942-dc8d9215ee47-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:17.620125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.620091    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:17.620629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.620612    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"]
Nov 03 03:45:17.621698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.621677    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg" pod "ca541e02-95bb-4474-a942-dc8d9215ee47" (UID: "ca541e02-95bb-4474-a942-dc8d9215ee47"). InnerVolumeSpecName "kube-api-access-64rtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:17.624625 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.624608    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894625-xtjkb"]
Nov 03 03:45:17.720571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.720515    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts\" (UniqueName: \"kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts\") pod \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\" (UID: \"5b0b2bfd-c218-42dc-bf6f-ec53120614b5\") "
Nov 03 03:45:17.721079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.721060    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rtg\" (UniqueName: \"kubernetes.io/projected/ca541e02-95bb-4474-a942-dc8d9215ee47-kube-api-access-64rtg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:17.722740 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.722720    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts" pod "5b0b2bfd-c218-42dc-bf6f-ec53120614b5" (UID: "5b0b2bfd-c218-42dc-bf6f-ec53120614b5"). InnerVolumeSpecName "kube-api-access-7lrts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:17.821636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:17.821607    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrts\" (UniqueName: \"kubernetes.io/projected/5b0b2bfd-c218-42dc-bf6f-ec53120614b5-kube-api-access-7lrts\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:18.347126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.347089    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"
Nov 03 03:45:18.347126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.347099    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j" event={"ID":"5b0b2bfd-c218-42dc-bf6f-ec53120614b5","Type":"ContainerDied","Data":"b5f9cc53f448349eef378bd609c86661679cb16027c206cdd572c54fbb2ffa66"}
Nov 03 03:45:18.347646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.347193    5515 scope.go:122] "RemoveContainer" containerID="5923bcadc960a6b5b3390cf20ce5d99e0a0ff075d9970aa50e6e5b258309ea29"
Nov 03 03:45:18.349538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.349522    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"
Nov 03 03:45:18.349664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.349598    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2" event={"ID":"ca541e02-95bb-4474-a942-dc8d9215ee47","Type":"ContainerDied","Data":"ccca2b269f978426d1b04da755df19c4630ef5b197186d6e9cdd9d46b3dc0e1f"}
Nov 03 03:45:18.359357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.359336    5515 scope.go:122] "RemoveContainer" containerID="857ed33220300ee688b3f531de21def32b1eb8e6d1e75889d053fa55f9c13d84"
Nov 03 03:45:18.502126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.502105    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:18.514358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.514334    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"]
Nov 03 03:45:18.523444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.523421    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-68b670-29894625-5m2c2"]
Nov 03 03:45:18.527795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.527772    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca\") pod \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") "
Nov 03 03:45:18.527968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.527955    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4\" (UniqueName: \"kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4\") pod \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\" (UID: \"c7f5a921-9dcb-4f82-a761-4a867b2e60c3\") "
Nov 03 03:45:18.528150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.528127    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca" pod "c7f5a921-9dcb-4f82-a761-4a867b2e60c3" (UID: "c7f5a921-9dcb-4f82-a761-4a867b2e60c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:18.528352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.528336    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:18.530618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.530589    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4" pod "c7f5a921-9dcb-4f82-a761-4a867b2e60c3" (UID: "c7f5a921-9dcb-4f82-a761-4a867b2e60c3"). InnerVolumeSpecName "kube-api-access-pxxm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:18.629593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.629542    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxxm4\" (UniqueName: \"kubernetes.io/projected/c7f5a921-9dcb-4f82-a761-4a867b2e60c3-kube-api-access-pxxm4\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:18.750025 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.749997    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"]
Nov 03 03:45:18.760468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.760443    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a9e5cd-29894625-95b8j"]
Nov 03 03:45:18.897030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.896958    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" path="/var/lib/kubelet/pods/5b0b2bfd-c218-42dc-bf6f-ec53120614b5/volumes"
Nov 03 03:45:18.897412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.897398    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" path="/var/lib/kubelet/pods/ca541e02-95bb-4474-a942-dc8d9215ee47/volumes"
Nov 03 03:45:18.897826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.897811    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" path="/var/lib/kubelet/pods/efc19370-fbe9-4490-bc0a-fb247bf11e81/volumes"
Nov 03 03:45:18.898988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:18.898974    5515 scope.go:122] "RemoveContainer" containerID="6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d"
Nov 03 03:45:19.302247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.302217    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:45:19.358003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.357974    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"
Nov 03 03:45:19.358003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.357991    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf" event={"ID":"c7f5a921-9dcb-4f82-a761-4a867b2e60c3","Type":"ContainerDied","Data":"590963ec33af3c37b52fbb874073c2327218a6a41e5a1815c4cefa22e91f5243"}
Nov 03 03:45:19.358509 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.358047    5515 scope.go:122] "RemoveContainer" containerID="2203e6c0feac02983b94453b3c23e90d74212ae985e127cd44b71483c8b8d0e1"
Nov 03 03:45:19.361280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.361257    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerStarted","Data":"e270296f2d97f2f35c5efa930ddb83dd857c703ed1055a31df1187c63fa490d4"}
Nov 03 03:45:19.654026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.653995    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"]
Nov 03 03:45:19.667200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:19.667169    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894625-t2lmf"]
Nov 03 03:45:20.371753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.371719    5515 generic.go:356] "Generic (PLEG): container finished" podID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerID="e270296f2d97f2f35c5efa930ddb83dd857c703ed1055a31df1187c63fa490d4" exitCode=60
Nov 03 03:45:20.372295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.371784    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerDied","Data":"e270296f2d97f2f35c5efa930ddb83dd857c703ed1055a31df1187c63fa490d4"}
Nov 03 03:45:20.372295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.371846    5515 scope.go:122] "RemoveContainer" containerID="6c943eaaa1ac773ce7d534b9495fc99223ecde6159222a292e3bfdc522533b4d"
Nov 03 03:45:20.372295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.372144    5515 scope.go:122] "RemoveContainer" containerID="e270296f2d97f2f35c5efa930ddb83dd857c703ed1055a31df1187c63fa490d4"
Nov 03 03:45:20.372462 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:20.372438    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv_odh-ai-gateway-infra(d88a5f21-071a-4853-bdd9-3e8aff037b2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e"
Nov 03 03:45:20.548003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.547976    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"]
Nov 03 03:45:20.894877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.894850    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" path="/var/lib/kubelet/pods/c7f5a921-9dcb-4f82-a761-4a867b2e60c3/volumes"
Nov 03 03:45:20.982852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:20.982822    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:20.984671 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:20.984631    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:45:21.534913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.534886    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:21.563471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.563441    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca\") pod \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") "
Nov 03 03:45:21.563624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.563508    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh\" (UniqueName: \"kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh\") pod \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\" (UID: \"d88a5f21-071a-4853-bdd9-3e8aff037b2e\") "
Nov 03 03:45:21.563772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.563751    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca" pod "d88a5f21-071a-4853-bdd9-3e8aff037b2e" (UID: "d88a5f21-071a-4853-bdd9-3e8aff037b2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:21.563833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.563818    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a5f21-071a-4853-bdd9-3e8aff037b2e-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:21.565647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.565628    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh" pod "d88a5f21-071a-4853-bdd9-3e8aff037b2e" (UID: "d88a5f21-071a-4853-bdd9-3e8aff037b2e"). InnerVolumeSpecName "kube-api-access-vsldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:21.665484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:21.665411    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsldh\" (UniqueName: \"kubernetes.io/projected/d88a5f21-071a-4853-bdd9-3e8aff037b2e-kube-api-access-vsldh\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:22.387979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:22.387951    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"
Nov 03 03:45:22.387979 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:22.387971    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv" event={"ID":"d88a5f21-071a-4853-bdd9-3e8aff037b2e","Type":"ContainerDied","Data":"5b407c681f13025dacb40f45622be5d76bd003e5dad4f901186721f3281f1251"}
Nov 03 03:45:22.388338 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:22.388040    5515 scope.go:122] "RemoveContainer" containerID="e270296f2d97f2f35c5efa930ddb83dd857c703ed1055a31df1187c63fa490d4"
Nov 03 03:45:22.561868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:22.561793    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"]
Nov 03 03:45:22.612671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:22.612640    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b1ad4e-29894625-7tktv"]
Nov 03 03:45:22.894059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:22.894027    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" path="/var/lib/kubelet/pods/d88a5f21-071a-4853-bdd9-3e8aff037b2e/volumes"
Nov 03 03:45:25.396784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.396756    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:25.603105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603071    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"]
Nov 03 03:45:25.603754 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603732    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.603754 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603753    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603773    5515 cpu_manager.go:414] "containerMap: removing container" podUID="734b45c9-9083-4105-890a-3a9f18a6494f" containerName="payload-processing"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603781    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b45c9-9083-4105-890a-3a9f18a6494f" containerName="payload-processing"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603800    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603806    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603819    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0c0d3818-5409-4e51-ab2f-296ae66a954c" containerName="payload-pre-processing"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603826    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0d3818-5409-4e51-ab2f-296ae66a954c" containerName="payload-pre-processing"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603848    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603854    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603891    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603899    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603911    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603919    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603928    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603935    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603946    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:45:25.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603953    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603969    5515 cpu_manager.go:414] "containerMap: removing container" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603977    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603986    5515 cpu_manager.go:414] "containerMap: removing container" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.603994    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604007    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604014    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604055    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604063    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604081    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604088    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604235    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604248    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604258    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604267    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604276    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604283    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604291    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604298    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604309    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604318    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604328    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5a921-9dcb-4f82-a761-4a867b2e60c3" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604337    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604347    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604357    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b45c9-9083-4105-890a-3a9f18a6494f" containerName="payload-processing"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604367    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0b2bfd-c218-42dc-bf6f-ec53120614b5" containerName="cleanup"
Nov 03 03:45:25.604794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.604378    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0d3818-5409-4e51-ab2f-296ae66a954c" containerName="payload-pre-processing"
Nov 03 03:45:25.609745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.609724    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.711748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.711680    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb58j\" (UniqueName: \"kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.711748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.711726    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.711925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.711871    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.711925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.711913    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.713698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.713680    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9b4cb2-serving-cert\""
Nov 03 03:45:25.716358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.716338    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"]
Nov 03 03:45:25.813594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.813533    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.813741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.813645    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.813741 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.813682    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.813826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.813785    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb58j\" (UniqueName: \"kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.813826 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:25.813803    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-serving-cert: secret "maas-api-e2e-praxis-b-9b4cb2-serving-cert" not found
Nov 03 03:45:25.813908 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:25.813873    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls podName:f590246a-4c24-48da-bed0-36bc2ef38d8b nodeName:}" failed. No retries permitted until 2026-11-03 03:45:26.313851639 +0000 UTC m=+2617.544744056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls") pod "maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" (UID: "f590246a-4c24-48da-bed0-36bc2ef38d8b") : secret "maas-api-e2e-praxis-b-9b4cb2-serving-cert" not found
Nov 03 03:45:25.814233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.814216    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.816215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.816198    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:25.882467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:25.882400    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb58j\" (UniqueName: \"kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:26.322745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:26.322704    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:26.325242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:26.325208    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:26.524088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:26.524054    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:26.903173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:26.903143    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"]
Nov 03 03:45:26.903813 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:45:26.903785    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf590246a_4c24_48da_bed0_36bc2ef38d8b.slice/crio-ef84691d67bceaa45e4b69f269143f60aa67176f65289c308940e4556bb65f7b WatchSource:0}: Error finding container ef84691d67bceaa45e4b69f269143f60aa67176f65289c308940e4556bb65f7b: Status 404 returned error can't find the container with id ef84691d67bceaa45e4b69f269143f60aa67176f65289c308940e4556bb65f7b
Nov 03 03:45:27.049815 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:27.049789    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:27.421486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:27.421457    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" event={"ID":"f590246a-4c24-48da-bed0-36bc2ef38d8b","Type":"ContainerStarted","Data":"ef84691d67bceaa45e4b69f269143f60aa67176f65289c308940e4556bb65f7b"}
Nov 03 03:45:28.429821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:28.429791    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" event={"ID":"f590246a-4c24-48da-bed0-36bc2ef38d8b","Type":"ContainerStarted","Data":"7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42"}
Nov 03 03:45:28.430215 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:28.429986    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:28.549543 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:28.549492    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" podStartSLOduration=2.744770212 podStartE2EDuration="3.54947931s" totalImagesPullingTime="804.709098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:28.548847547 +0000 UTC m=+2619.779739966" watchObservedRunningTime="2026-11-03 03:45:28.54947931 +0000 UTC m=+2619.780371727"
Nov 03 03:45:31.169507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:31.169469    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:32.433470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:32.433438    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:34.443059 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:34.443034    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:45:36.361789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:36.361756    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:38.591002 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:38.590966    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:41.332259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:41.332223    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:45.602053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:45.602018    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:47.434835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:47.434792    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:50.742305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:50.742254    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:50.889448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:50.889364    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:50.984814 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:50.984781    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:45:51.671921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:51.671884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:52.018784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.018699    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"]
Nov 03 03:45:52.019300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.018907    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj" podUID="abc45778-8f2e-4ff7-b200-5233d868dbe9" containerName="payload-processing" containerID="cri-o://5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995" gracePeriod=30
Nov 03 03:45:52.126853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.126816    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"]
Nov 03 03:45:52.127088 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.127063    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4" podUID="d13f8abc-7373-455b-b065-5262e76c5949" containerName="payload-pre-processing" containerID="cri-o://b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0" gracePeriod=30
Nov 03 03:45:52.178732 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.178679    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:52.315040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.315011    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:45:52.323818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.323787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:52.344515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.344491    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw\" (UniqueName: \"kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw\") pod \"abc45778-8f2e-4ff7-b200-5233d868dbe9\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") "
Nov 03 03:45:52.344703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.344574    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume\") pod \"abc45778-8f2e-4ff7-b200-5233d868dbe9\" (UID: \"abc45778-8f2e-4ff7-b200-5233d868dbe9\") "
Nov 03 03:45:52.345049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.345019    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume" pod "abc45778-8f2e-4ff7-b200-5233d868dbe9" (UID: "abc45778-8f2e-4ff7-b200-5233d868dbe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:52.346771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.346747    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw" pod "abc45778-8f2e-4ff7-b200-5233d868dbe9" (UID: "abc45778-8f2e-4ff7-b200-5233d868dbe9"). InnerVolumeSpecName "kube-api-access-cmbrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:52.409240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.409212    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:45:52.446214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.446182    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume\") pod \"d13f8abc-7373-455b-b065-5262e76c5949\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") "
Nov 03 03:45:52.446408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.446259    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj\" (UniqueName: \"kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj\") pod \"d13f8abc-7373-455b-b065-5262e76c5949\" (UID: \"d13f8abc-7373-455b-b065-5262e76c5949\") "
Nov 03 03:45:52.446588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.446528    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume" pod "d13f8abc-7373-455b-b065-5262e76c5949" (UID: "d13f8abc-7373-455b-b065-5262e76c5949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:45:52.446688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.446619    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc45778-8f2e-4ff7-b200-5233d868dbe9-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:52.446688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.446636    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbrw\" (UniqueName: \"kubernetes.io/projected/abc45778-8f2e-4ff7-b200-5233d868dbe9-kube-api-access-cmbrw\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:52.448680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.448655    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj" pod "d13f8abc-7373-455b-b065-5262e76c5949" (UID: "d13f8abc-7373-455b-b065-5262e76c5949"). InnerVolumeSpecName "kube-api-access-hkfgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:45:52.557714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.547607    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13f8abc-7373-455b-b065-5262e76c5949-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:52.557714 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.547650    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfgj\" (UniqueName: \"kubernetes.io/projected/d13f8abc-7373-455b-b065-5262e76c5949-kube-api-access-hkfgj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:45:52.602604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.598931    5515 generic.go:356] "Generic (PLEG): container finished" podID="abc45778-8f2e-4ff7-b200-5233d868dbe9" containerID="5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995" exitCode=0
Nov 03 03:45:52.602604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.599050    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj" event={"ID":"abc45778-8f2e-4ff7-b200-5233d868dbe9","Type":"ContainerDied","Data":"5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995"}
Nov 03 03:45:52.602604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.599096    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj" event={"ID":"abc45778-8f2e-4ff7-b200-5233d868dbe9","Type":"ContainerDied","Data":"6544abf30102e1cb119b1577422d1e3f8d3706b79c3c96b4280d32afa7274655"}
Nov 03 03:45:52.602604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.599118    5515 scope.go:122] "RemoveContainer" containerID="5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995"
Nov 03 03:45:52.602604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.599249    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"
Nov 03 03:45:52.607255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.607200    5515 generic.go:356] "Generic (PLEG): container finished" podID="d13f8abc-7373-455b-b065-5262e76c5949" containerID="b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0" exitCode=0
Nov 03 03:45:52.607412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.607399    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4" event={"ID":"d13f8abc-7373-455b-b065-5262e76c5949","Type":"ContainerDied","Data":"b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0"}
Nov 03 03:45:52.607489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.607449    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4" event={"ID":"d13f8abc-7373-455b-b065-5262e76c5949","Type":"ContainerDied","Data":"5296e8b09029d77fc80bd659039facc30b7f13e075e4529049a61ca2b92f56ee"}
Nov 03 03:45:52.607594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.607578    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"
Nov 03 03:45:52.624523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.624414    5515 scope.go:122] "RemoveContainer" containerID="5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995"
Nov 03 03:45:52.624828 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:52.624796    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995\": container with ID starting with 5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995 not found: ID does not exist" containerID="5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995"
Nov 03 03:45:52.624947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.624839    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995"} err="failed to get container status \"5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995\": rpc error: code = NotFound desc = could not find container \"5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995\": container with ID starting with 5fb59654faf0eb786059ebe252c5b07ac464e06338cef986672cc7abe9c5b995 not found: ID does not exist"
Nov 03 03:45:52.624947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.624864    5515 scope.go:122] "RemoveContainer" containerID="b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0"
Nov 03 03:45:52.635367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.635347    5515 scope.go:122] "RemoveContainer" containerID="b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0"
Nov 03 03:45:52.635711 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:45:52.635688    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0\": container with ID starting with b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0 not found: ID does not exist" containerID="b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0"
Nov 03 03:45:52.635787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.635721    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0"} err="failed to get container status \"b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0\": rpc error: code = NotFound desc = could not find container \"b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0\": container with ID starting with b676fce0b2b20e50b0cf14d3e7f0659b0d96e00894bc88c52188be6d0f3784a0 not found: ID does not exist"
Nov 03 03:45:52.772152 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.772119    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"]
Nov 03 03:45:52.778142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.778116    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a6b9ff-5b54467cdb-jc4mj"]
Nov 03 03:45:52.894602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.894571    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc45778-8f2e-4ff7-b200-5233d868dbe9" path="/var/lib/kubelet/pods/abc45778-8f2e-4ff7-b200-5233d868dbe9/volumes"
Nov 03 03:45:52.894888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.894875    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"]
Nov 03 03:45:52.894940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:52.894905    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b9ff-7bcb56d9ff-sdqh4"]
Nov 03 03:45:53.469468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:53.469435    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:53.616747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:53.616716    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:53.770459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:53.770381    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:54.893858 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:54.893826    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13f8abc-7373-455b-b065-5262e76c5949" path="/var/lib/kubelet/pods/d13f8abc-7373-455b-b065-5262e76c5949/volumes"
Nov 03 03:45:55.025642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:55.025610    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:55.170629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:55.170493    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:55.638202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:55.638172    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:56.439122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:56.439091    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:56.584407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:56.584377    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:57.072645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:57.072616    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"]
Nov 03 03:45:57.072850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:57.072830    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy" containerID="cri-o://414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4" gracePeriod=30
Nov 03 03:45:57.084006 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:57.083980    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:57.516340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:57.516291    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:57.830951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:57.830856    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:58.926388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:58.926357    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:45:59.420442 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:45:59.420373    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:02.342898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.342838    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:46:02.467514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467479    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.467514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467515    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.467763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467536    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.467809 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467780    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.467885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467866    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:46:02.467928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467883    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.467928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467886    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:46:02.467928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467918    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.468075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.467965    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.468075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468026    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.468075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468041    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:46:02.468244 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468131    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo\") pod \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\" (UID: \"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c\") "
Nov 03 03:46:02.468368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468346    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:02.468475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468419    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:46:02.468833 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468810    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.468926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468840    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.468926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468856    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.468926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468870    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.468926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.468884    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.470179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.470143    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "kube-api-access-654zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:02.470179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.470161    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:02.470366 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.470348    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:46:02.470426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.470367    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo" pod "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" (UID: "0aa7c9b5-972c-4d09-b22d-b60f725f1f8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:46:02.545709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.545676    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:02.569929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.569902    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.569929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.569926    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654zd\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-kube-api-access-654zd\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.570094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.569937    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.570094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.569946    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:02.685062 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.685023    5515 generic.go:356] "Generic (PLEG): container finished" podID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerID="414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4" exitCode=0
Nov 03 03:46:02.685270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.685083    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"
Nov 03 03:46:02.685270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.685075    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" event={"ID":"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c","Type":"ContainerDied","Data":"414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4"}
Nov 03 03:46:02.685270 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.685242    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr" event={"ID":"0aa7c9b5-972c-4d09-b22d-b60f725f1f8c","Type":"ContainerDied","Data":"dcd771e1ed356de3010ddfe2e2aa41a797920defe37a49464557a20a03b1bbae"}
Nov 03 03:46:02.685461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.685281    5515 scope.go:122] "RemoveContainer" containerID="414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4"
Nov 03 03:46:02.701413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.701392    5515 scope.go:122] "RemoveContainer" containerID="f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"
Nov 03 03:46:02.710331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.710312    5515 scope.go:122] "RemoveContainer" containerID="414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4"
Nov 03 03:46:02.710638 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:02.710613    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4\": container with ID starting with 414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4 not found: ID does not exist" containerID="414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4"
Nov 03 03:46:02.710712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.710652    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4"} err="failed to get container status \"414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4\": rpc error: code = NotFound desc = could not find container \"414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4\": container with ID starting with 414ad93d2335612f3c993af0917f1459d9ab78860d0173e3ef7a29f88b137df4 not found: ID does not exist"
Nov 03 03:46:02.710712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.710678    5515 scope.go:122] "RemoveContainer" containerID="f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"
Nov 03 03:46:02.710955 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:02.710932    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896\": container with ID starting with f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896 not found: ID does not exist" containerID="f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"
Nov 03 03:46:02.711041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.710958    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896"} err="failed to get container status \"f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896\": rpc error: code = NotFound desc = could not find container \"f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896\": container with ID starting with f8f5002fcc12be028fc0d8cdc310459c250d54ebaa0e25b56e427168fbe18896 not found: ID does not exist"
Nov 03 03:46:02.855454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.855424    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"]
Nov 03 03:46:02.859846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.859823    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-754aeb-openshift-default-7947d5d74f-nf6fr"]
Nov 03 03:46:02.888255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.888230    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:02.897469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.897432    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" path="/var/lib/kubelet/pods/0aa7c9b5-972c-4d09-b22d-b60f725f1f8c/volumes"
Nov 03 03:46:02.965044 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:02.964967    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:46:03.348382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.348291    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"]
Nov 03 03:46:03.348854 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.348582    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" podUID="6bddcb24-f783-4859-9b74-98bd5404b9a2" containerName="maas-api" containerID="cri-o://f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040" gracePeriod=30
Nov 03 03:46:03.349506 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.349480    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:03.493997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.493966    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:03.666846 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.666821    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:46:03.693049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.693022    5515 generic.go:356] "Generic (PLEG): container finished" podID="6bddcb24-f783-4859-9b74-98bd5404b9a2" containerID="f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040" exitCode=0
Nov 03 03:46:03.693217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.693077    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"
Nov 03 03:46:03.693217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.693112    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" event={"ID":"6bddcb24-f783-4859-9b74-98bd5404b9a2","Type":"ContainerDied","Data":"f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040"}
Nov 03 03:46:03.693217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.693166    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb" event={"ID":"6bddcb24-f783-4859-9b74-98bd5404b9a2","Type":"ContainerDied","Data":"dd86b13d3e2d47433d797166a0c70a1c84602f6ecc971b4622e81f4f2f3b5975"}
Nov 03 03:46:03.693217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.693196    5515 scope.go:122] "RemoveContainer" containerID="f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040"
Nov 03 03:46:03.702532 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.702514    5515 scope.go:122] "RemoveContainer" containerID="f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040"
Nov 03 03:46:03.702823 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:03.702803    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040\": container with ID starting with f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040 not found: ID does not exist" containerID="f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040"
Nov 03 03:46:03.702896 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.702833    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040"} err="failed to get container status \"f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040\": rpc error: code = NotFound desc = could not find container \"f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040\": container with ID starting with f943ada2296e252a1129627277296936b6f3c714fe8a08ab00ded9a2a6b24040 not found: ID does not exist"
Nov 03 03:46:03.789881 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.789849    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca\") pod \"6bddcb24-f783-4859-9b74-98bd5404b9a2\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") "
Nov 03 03:46:03.790077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.789897    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv\" (UniqueName: \"kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv\") pod \"6bddcb24-f783-4859-9b74-98bd5404b9a2\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") "
Nov 03 03:46:03.790077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.790060    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs\") pod \"6bddcb24-f783-4859-9b74-98bd5404b9a2\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") "
Nov 03 03:46:03.790197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.790097    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\") pod \"6bddcb24-f783-4859-9b74-98bd5404b9a2\" (UID: \"6bddcb24-f783-4859-9b74-98bd5404b9a2\") "
Nov 03 03:46:03.790304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.790278    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca" pod "6bddcb24-f783-4859-9b74-98bd5404b9a2" (UID: "6bddcb24-f783-4859-9b74-98bd5404b9a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:03.790608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.790589    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bddcb24-f783-4859-9b74-98bd5404b9a2-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:03.792710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.792676    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv" pod "6bddcb24-f783-4859-9b74-98bd5404b9a2" (UID: "6bddcb24-f783-4859-9b74-98bd5404b9a2"). InnerVolumeSpecName "kube-api-access-sdvsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:03.792835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.792755    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls" pod "6bddcb24-f783-4859-9b74-98bd5404b9a2" (UID: "6bddcb24-f783-4859-9b74-98bd5404b9a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:46:03.793402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.793375    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs" pod "6bddcb24-f783-4859-9b74-98bd5404b9a2" (UID: "6bddcb24-f783-4859-9b74-98bd5404b9a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:46:03.860209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.860177    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"]
Nov 03 03:46:03.860391 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.860372    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww" podUID="db146775-1c92-47fd-9f0a-1e29e00e82fd" containerName="payload-pre-processing" containerID="cri-o://41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1" gracePeriod=30
Nov 03 03:46:03.891482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.891426    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:03.891482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.891454    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bddcb24-f783-4859-9b74-98bd5404b9a2-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:03.891482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:03.891466    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvsv\" (UniqueName: \"kubernetes.io/projected/6bddcb24-f783-4859-9b74-98bd5404b9a2-kube-api-access-sdvsv\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:04.131109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.128503    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:04.135197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.135159    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"]
Nov 03 03:46:04.141661 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.141591    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b9c10a-664f499fb6-qqgrb"]
Nov 03 03:46:04.207192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.207160    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:46:04.288371 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.288324    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:04.328131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.328095    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx\" (UniqueName: \"kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx\") pod \"db146775-1c92-47fd-9f0a-1e29e00e82fd\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") "
Nov 03 03:46:04.328327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.328211    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume\") pod \"db146775-1c92-47fd-9f0a-1e29e00e82fd\" (UID: \"db146775-1c92-47fd-9f0a-1e29e00e82fd\") "
Nov 03 03:46:04.328643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.328614    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume" pod "db146775-1c92-47fd-9f0a-1e29e00e82fd" (UID: "db146775-1c92-47fd-9f0a-1e29e00e82fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:04.328856 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.328833    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db146775-1c92-47fd-9f0a-1e29e00e82fd-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:04.331083 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.331062    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx" pod "db146775-1c92-47fd-9f0a-1e29e00e82fd" (UID: "db146775-1c92-47fd-9f0a-1e29e00e82fd"). InnerVolumeSpecName "kube-api-access-w6tbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:04.430402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.430326    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tbx\" (UniqueName: \"kubernetes.io/projected/db146775-1c92-47fd-9f0a-1e29e00e82fd-kube-api-access-w6tbx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:04.702926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.702827    5515 generic.go:356] "Generic (PLEG): container finished" podID="db146775-1c92-47fd-9f0a-1e29e00e82fd" containerID="41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1" exitCode=0
Nov 03 03:46:04.702926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.702876    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"
Nov 03 03:46:04.702926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.702908    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww" event={"ID":"db146775-1c92-47fd-9f0a-1e29e00e82fd","Type":"ContainerDied","Data":"41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1"}
Nov 03 03:46:04.703232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.702958    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww" event={"ID":"db146775-1c92-47fd-9f0a-1e29e00e82fd","Type":"ContainerDied","Data":"2dd8024948748fcdde1b7c986f97641c3ef74b9b13f8584c6534f2c5a8bb313b"}
Nov 03 03:46:04.703232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.702983    5515 scope.go:122] "RemoveContainer" containerID="41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1"
Nov 03 03:46:04.735781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.726878    5515 scope.go:122] "RemoveContainer" containerID="41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1"
Nov 03 03:46:04.735781 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:04.727290    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1\": container with ID starting with 41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1 not found: ID does not exist" containerID="41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1"
Nov 03 03:46:04.735781 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.727328    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1"} err="failed to get container status \"41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1\": rpc error: code = NotFound desc = could not find container \"41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1\": container with ID starting with 41ffd389aa610c0b08afac82f0d7757fbc6286d5e4b7ffdd8d3542d7d84651b1 not found: ID does not exist"
Nov 03 03:46:04.825598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.825543    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:04.897279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.897245    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bddcb24-f783-4859-9b74-98bd5404b9a2" path="/var/lib/kubelet/pods/6bddcb24-f783-4859-9b74-98bd5404b9a2/volumes"
Nov 03 03:46:04.897635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.897615    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"]
Nov 03 03:46:04.897745 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:04.897654    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b9c10a-6cff4fbf6c-vd5ww"]
Nov 03 03:46:05.508694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:05.508665    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:06.058969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:06.058937    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:06.692594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:06.691221    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"]
Nov 03 03:46:06.790758 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:06.790727    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"]
Nov 03 03:46:06.896924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:06.896888    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db146775-1c92-47fd-9f0a-1e29e00e82fd" path="/var/lib/kubelet/pods/db146775-1c92-47fd-9f0a-1e29e00e82fd/volumes"
Nov 03 03:46:07.046587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:07.046488    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:07.191504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:07.191471    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:08.688032 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:08.687991    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:08.838735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:08.838699    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:10.192983 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:10.192950    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:10.644231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:10.644197    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:10.813771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:10.813672    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:10.984023 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:10.983930    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:11.692169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:11.692118    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m" podUID="c653fd77-8aa7-4af3-a244-80a1a8c135c2" containerName="payload-processing" containerID="cri-o://ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e" gracePeriod=25
Nov 03 03:46:11.791629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:11.791592    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn" podUID="a1baa566-ebed-4a3d-8232-c6cbd115d30e" containerName="payload-pre-processing" containerID="cri-o://ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8" gracePeriod=25
Nov 03 03:46:11.810242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:11.810219    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:11.963374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:11.963310    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:46:12.048862 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.048841    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:46:12.149602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.149570    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj\" (UniqueName: \"kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj\") pod \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") "
Nov 03 03:46:12.149847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.149619    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume\") pod \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\" (UID: \"c653fd77-8aa7-4af3-a244-80a1a8c135c2\") "
Nov 03 03:46:12.149847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.149713    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq\" (UniqueName: \"kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq\") pod \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") "
Nov 03 03:46:12.149847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.149815    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume\") pod \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\" (UID: \"a1baa566-ebed-4a3d-8232-c6cbd115d30e\") "
Nov 03 03:46:12.150094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.150071    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume" pod "c653fd77-8aa7-4af3-a244-80a1a8c135c2" (UID: "c653fd77-8aa7-4af3-a244-80a1a8c135c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:12.150256 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.150226    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume" pod "a1baa566-ebed-4a3d-8232-c6cbd115d30e" (UID: "a1baa566-ebed-4a3d-8232-c6cbd115d30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:12.150327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.150291    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c653fd77-8aa7-4af3-a244-80a1a8c135c2-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:12.151965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.151936    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj" pod "c653fd77-8aa7-4af3-a244-80a1a8c135c2" (UID: "c653fd77-8aa7-4af3-a244-80a1a8c135c2"). InnerVolumeSpecName "kube-api-access-b5dmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:12.152102 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.152084    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq" pod "a1baa566-ebed-4a3d-8232-c6cbd115d30e" (UID: "a1baa566-ebed-4a3d-8232-c6cbd115d30e"). InnerVolumeSpecName "kube-api-access-286wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:12.182605 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.182575    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:12.251698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.251617    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5dmj\" (UniqueName: \"kubernetes.io/projected/c653fd77-8aa7-4af3-a244-80a1a8c135c2-kube-api-access-b5dmj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:12.251698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.251653    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286wq\" (UniqueName: \"kubernetes.io/projected/a1baa566-ebed-4a3d-8232-c6cbd115d30e-kube-api-access-286wq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:12.251698 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.251671    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1baa566-ebed-4a3d-8232-c6cbd115d30e-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:12.767028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.766994    5515 generic.go:356] "Generic (PLEG): container finished" podID="c653fd77-8aa7-4af3-a244-80a1a8c135c2" containerID="ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e" exitCode=0
Nov 03 03:46:12.767580 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.767192    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m" event={"ID":"c653fd77-8aa7-4af3-a244-80a1a8c135c2","Type":"ContainerDied","Data":"ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e"}
Nov 03 03:46:12.767703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.767604    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m" event={"ID":"c653fd77-8aa7-4af3-a244-80a1a8c135c2","Type":"ContainerDied","Data":"a50e184a1ff71dcb63309f38aed6ccb67c2fe1b6880369294bb33e10173b2d32"}
Nov 03 03:46:12.767703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.767639    5515 scope.go:122] "RemoveContainer" containerID="ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e"
Nov 03 03:46:12.767835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.767275    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"
Nov 03 03:46:12.779722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.771270    5515 generic.go:356] "Generic (PLEG): container finished" podID="a1baa566-ebed-4a3d-8232-c6cbd115d30e" containerID="ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8" exitCode=0
Nov 03 03:46:12.779722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.771396    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn" event={"ID":"a1baa566-ebed-4a3d-8232-c6cbd115d30e","Type":"ContainerDied","Data":"ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8"}
Nov 03 03:46:12.779722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.771436    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn" event={"ID":"a1baa566-ebed-4a3d-8232-c6cbd115d30e","Type":"ContainerDied","Data":"9590d2b6ec035ec90f2cccb1bc385f959b9d23c668eb65b6dfb4eeb43230b6e7"}
Nov 03 03:46:12.779722 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.771501    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"
Nov 03 03:46:12.798180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.798160    5515 scope.go:122] "RemoveContainer" containerID="ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e"
Nov 03 03:46:12.802145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.801055    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:12.802145 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:12.801429    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e\": container with ID starting with ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e not found: ID does not exist" containerID="ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e"
Nov 03 03:46:12.802145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.801463    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e"} err="failed to get container status \"ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e\": rpc error: code = NotFound desc = could not find container \"ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e\": container with ID starting with ff868aa2230e1924d81569614997fb0d90ee0c8279784099850272be0470dd6e not found: ID does not exist"
Nov 03 03:46:12.802145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.801491    5515 scope.go:122] "RemoveContainer" containerID="ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8"
Nov 03 03:46:12.815642 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.815619    5515 scope.go:122] "RemoveContainer" containerID="ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8"
Nov 03 03:46:12.815986 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:12.815960    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8\": container with ID starting with ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8 not found: ID does not exist" containerID="ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8"
Nov 03 03:46:12.816079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.816000    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8"} err="failed to get container status \"ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8\": rpc error: code = NotFound desc = could not find container \"ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8\": container with ID starting with ab4ce9c1c7922844e835dc480f11675e265036c7f7303cac0d2708ca229f25f8 not found: ID does not exist"
Nov 03 03:46:12.935927 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.935887    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"]
Nov 03 03:46:12.941877 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:12.941853    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b755ee-cfffb74b9-p4s9m"]
Nov 03 03:46:13.069036 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.069008    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:13.112719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.112681    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"]
Nov 03 03:46:13.132519 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.132491    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b755ee-cf4fd688c-47vmn"]
Nov 03 03:46:13.780237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.780200    5515 generic.go:356] "Generic (PLEG): container finished" podID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerID="42d8efa9f0ab2c9247a98fe6784ea9e7c4fcb00dc93700e08bab4a14ed48b43c" exitCode=0
Nov 03 03:46:13.780692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.780244    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" event={"ID":"509c0d87-f3b2-40e8-a21c-0ca99d6ae651","Type":"ContainerDied","Data":"42d8efa9f0ab2c9247a98fe6784ea9e7c4fcb00dc93700e08bab4a14ed48b43c"}
Nov 03 03:46:13.780692 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.780647    5515 scope.go:122] "RemoveContainer" containerID="42d8efa9f0ab2c9247a98fe6784ea9e7c4fcb00dc93700e08bab4a14ed48b43c"
Nov 03 03:46:13.782068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.782040    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:46:13.782144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.782116    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:46:13.782181 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:13.782148    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:46:14.793966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:14.793925    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" event={"ID":"509c0d87-f3b2-40e8-a21c-0ca99d6ae651","Type":"ContainerStarted","Data":"2b8095a16e051ff3ecd4d6a428f465dc4506b5c10cdc45f642bec7478bce25d3"}
Nov 03 03:46:14.894507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:14.894472    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1baa566-ebed-4a3d-8232-c6cbd115d30e" path="/var/lib/kubelet/pods/a1baa566-ebed-4a3d-8232-c6cbd115d30e/volumes"
Nov 03 03:46:14.895045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:14.895025    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c653fd77-8aa7-4af3-a244-80a1a8c135c2" path="/var/lib/kubelet/pods/c653fd77-8aa7-4af3-a244-80a1a8c135c2/volumes"
Nov 03 03:46:15.097133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:15.097032    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:46:15.097308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:15.097196    5515 patch_prober.go:29] interesting pod/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 03:46:15.097308 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:15.097237    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 03:46:15.914715 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:15.914683    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:16.096514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:16.096481    5515 patch_prober.go:29] interesting pod/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 03:46:16.096699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:16.096531    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 03:46:16.698565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:16.698525    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:16.908356 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:16.908327    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:17.058168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:17.058080    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:17.097728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:17.097694    5515 patch_prober.go:29] interesting pod/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 03:46:17.097890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:17.097748    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 03:46:18.096561 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:18.096522    5515 patch_prober.go:29] interesting pod/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 03:46:18.096967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:18.096615    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 03:46:19.096272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:19.096238    5515 patch_prober.go:29] interesting pod/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 03:46:19.096470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:19.096301    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 03:46:19.945929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:19.945885    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:20.096838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:20.096803    5515 patch_prober.go:29] interesting pod/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 03:46:20.097029 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:20.096865    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 03:46:20.119397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:20.119366    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:20.807853 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:20.807823    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:46:20.984448 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:20.984403    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:46:21.100713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:21.100637    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:46:21.101544 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:21.101528    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:46:21.283977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:21.283927    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:25.794228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:25.794167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:26.630400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:26.630368    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:27.022773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:27.022686    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:27.663475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:27.663447    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:27.862109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:27.862076    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:31.596478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:31.596446    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:31.819514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:31.819484    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:32.009632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:32.009595    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:32.156562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:32.156528    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:32.683051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:32.683022    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:46:35.102066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.102034    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:35.275337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.275304    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:35.515331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515298    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"]
Nov 03 03:46:35.515827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515813    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d13f8abc-7373-455b-b065-5262e76c5949" containerName="payload-pre-processing"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515828    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13f8abc-7373-455b-b065-5262e76c5949" containerName="payload-pre-processing"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515842    5515 cpu_manager.go:414] "containerMap: removing container" podUID="abc45778-8f2e-4ff7-b200-5233d868dbe9" containerName="payload-processing"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515847    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc45778-8f2e-4ff7-b200-5233d868dbe9" containerName="payload-processing"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515861    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515866    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515873    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6bddcb24-f783-4859-9b74-98bd5404b9a2" containerName="maas-api"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515878    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bddcb24-f783-4859-9b74-98bd5404b9a2" containerName="maas-api"
Nov 03 03:46:35.515894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515896    5515 cpu_manager.go:414] "containerMap: removing container" podUID="db146775-1c92-47fd-9f0a-1e29e00e82fd" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515901    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db146775-1c92-47fd-9f0a-1e29e00e82fd" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515910    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515914    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88a5f21-071a-4853-bdd9-3e8aff037b2e" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515921    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a1baa566-ebed-4a3d-8232-c6cbd115d30e" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515925    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1baa566-ebed-4a3d-8232-c6cbd115d30e" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515932    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515935    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515943    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515947    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515955    5515 cpu_manager.go:414] "containerMap: removing container" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515959    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc19370-fbe9-4490-bc0a-fb247bf11e81" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515964    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c653fd77-8aa7-4af3-a244-80a1a8c135c2" containerName="payload-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.515968    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c653fd77-8aa7-4af3-a244-80a1a8c135c2" containerName="payload-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516059    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="db146775-1c92-47fd-9f0a-1e29e00e82fd" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516068    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1baa566-ebed-4a3d-8232-c6cbd115d30e" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516075    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516082    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c653fd77-8aa7-4af3-a244-80a1a8c135c2" containerName="payload-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516089    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca541e02-95bb-4474-a942-dc8d9215ee47" containerName="cleanup"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516096    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13f8abc-7373-455b-b065-5262e76c5949" containerName="payload-pre-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516102    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc45778-8f2e-4ff7-b200-5233d868dbe9" containerName="payload-processing"
Nov 03 03:46:35.516186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.516111    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bddcb24-f783-4859-9b74-98bd5404b9a2" containerName="maas-api"
Nov 03 03:46:35.520162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.520146    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.574323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.574298    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e84ee2\""
Nov 03 03:46:35.577802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.577771    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"]
Nov 03 03:46:35.625440 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.625406    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e84ee2-dockercfg-8t59g\""
Nov 03 03:46:35.643456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.643415    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.643704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.643472    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjx8z\" (UniqueName: \"kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z\") pod \"payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.666467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.666434    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"]
Nov 03 03:46:35.667201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.667183    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa7c9b5-972c-4d09-b22d-b60f725f1f8c" containerName="istio-proxy"
Nov 03 03:46:35.670294 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.670278    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:35.727760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.727732    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"]
Nov 03 03:46:35.745418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.745386    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.745592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.745435    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjx8z\" (UniqueName: \"kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z\") pod \"payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.746105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.746081    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.807972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.807904    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjx8z\" (UniqueName: \"kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z\") pod \"payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.833609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.833587    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:35.846490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.846461    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxv2\" (UniqueName: \"kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2\") pod \"payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:35.846643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.846500    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:35.947926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.947893    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxv2\" (UniqueName: \"kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2\") pod \"payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:35.948097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.947955    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:35.948951 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:35.948927    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:36.009948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.009920    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxv2\" (UniqueName: \"kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2\") pod \"payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:36.208884 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:46:36.208848    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96cc613_0061_4097_8491_29ed21891d63.slice/crio-31b69929097ae9daa8e576b5ab12dc2c01cabd37c2611ae5bce16beb073a66ba WatchSource:0}: Error finding container 31b69929097ae9daa8e576b5ab12dc2c01cabd37c2611ae5bce16beb073a66ba: Status 404 returned error can't find the container with id 31b69929097ae9daa8e576b5ab12dc2c01cabd37c2611ae5bce16beb073a66ba
Nov 03 03:46:36.209297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.208964    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"]
Nov 03 03:46:36.285240 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.285211    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:36.670194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.670164    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"]
Nov 03 03:46:36.672049 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:46:36.672026    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c1f323_95e3_47ff_90f7_b41f093862fe.slice/crio-16a80eea6bd9cb1c849f66a7e7f8ae1f36cc12a950ea11497de2e4028ed93796 WatchSource:0}: Error finding container 16a80eea6bd9cb1c849f66a7e7f8ae1f36cc12a950ea11497de2e4028ed93796: Status 404 returned error can't find the container with id 16a80eea6bd9cb1c849f66a7e7f8ae1f36cc12a950ea11497de2e4028ed93796
Nov 03 03:46:36.915326 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.915292    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:36.922411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.922334    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc" event={"ID":"f96cc613-0061-4097-8491-29ed21891d63","Type":"ContainerStarted","Data":"0c1778b17e33f31011b541207463b4c77b52c5a4852e7f88b8ffbc064237edfd"}
Nov 03 03:46:36.922411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.922388    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc" event={"ID":"f96cc613-0061-4097-8491-29ed21891d63","Type":"ContainerStarted","Data":"31b69929097ae9daa8e576b5ab12dc2c01cabd37c2611ae5bce16beb073a66ba"}
Nov 03 03:46:36.922610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.922544    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:36.923940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.923920    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz" event={"ID":"58c1f323-95e3-47ff-90f7-b41f093862fe","Type":"ContainerStarted","Data":"10a56bbf2d6cf772d7ec36d8d81c23ca65e09435afb06bdd64d414edb13484df"}
Nov 03 03:46:36.924009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.923956    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz" event={"ID":"58c1f323-95e3-47ff-90f7-b41f093862fe","Type":"ContainerStarted","Data":"16a80eea6bd9cb1c849f66a7e7f8ae1f36cc12a950ea11497de2e4028ed93796"}
Nov 03 03:46:36.924009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:36.923976    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:37.042395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:37.042345    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc" podStartSLOduration=2.042332065 podStartE2EDuration="2.042332065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:37.04063956 +0000 UTC m=+2688.271531979" watchObservedRunningTime="2026-11-03 03:46:37.042332065 +0000 UTC m=+2688.273224521"
Nov 03 03:46:37.165582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:37.158891    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz" podStartSLOduration=2.158872558 podStartE2EDuration="2.158872558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:46:37.158642051 +0000 UTC m=+2688.389534470" watchObservedRunningTime="2026-11-03 03:46:37.158872558 +0000 UTC m=+2688.389764976"
Nov 03 03:46:41.471000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:41.470967    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:41.619076 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:41.619039    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:43.424953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:43.424916    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:45.105845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:45.105813    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:45.258332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:45.258275    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:46.797629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:46.797597    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:46.942324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:46.942293    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:47.336957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:47.336907    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:47.932093 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:47.932066    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:46:47.932492 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:47.932130    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:46:48.105780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:48.105746    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:48.627331 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:48.627296    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:50.091630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:50.091603    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:46:50.093864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:50.093843    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:46:50.124337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:50.124314    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:46:50.126490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:50.126474    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:46:50.984957 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:50.984859    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:46:53.833895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:53.833820    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"]
Nov 03 03:46:53.940255 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:53.940226    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"]
Nov 03 03:46:53.952955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:53.952921    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:54.187001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:54.186962    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:54.330983 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:54.330941    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:54.733707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:54.733670    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:58.340839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.340800    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"]
Nov 03 03:46:58.341237 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.341013    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" podUID="8d45858c-016d-472c-9c8b-b887d2278878" containerName="maas-api" containerID="cri-o://01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779" gracePeriod=30
Nov 03 03:46:58.480267 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.480224    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:46:58.627674 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.627648    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:46:58.711654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.711621    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\") pod \"8d45858c-016d-472c-9c8b-b887d2278878\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") "
Nov 03 03:46:58.711829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.711725    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca\") pod \"8d45858c-016d-472c-9c8b-b887d2278878\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") "
Nov 03 03:46:58.711829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.711805    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs\") pod \"8d45858c-016d-472c-9c8b-b887d2278878\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") "
Nov 03 03:46:58.711914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.711865    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh\" (UniqueName: \"kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh\") pod \"8d45858c-016d-472c-9c8b-b887d2278878\" (UID: \"8d45858c-016d-472c-9c8b-b887d2278878\") "
Nov 03 03:46:58.712141 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.712118    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca" pod "8d45858c-016d-472c-9c8b-b887d2278878" (UID: "8d45858c-016d-472c-9c8b-b887d2278878"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:58.712304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.712289    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d45858c-016d-472c-9c8b-b887d2278878-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:58.714011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.713982    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls" pod "8d45858c-016d-472c-9c8b-b887d2278878" (UID: "8d45858c-016d-472c-9c8b-b887d2278878"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:46:58.714110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.714061    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh" pod "8d45858c-016d-472c-9c8b-b887d2278878" (UID: "8d45858c-016d-472c-9c8b-b887d2278878"). InnerVolumeSpecName "kube-api-access-q99hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:58.714110 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.714102    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs" pod "8d45858c-016d-472c-9c8b-b887d2278878" (UID: "8d45858c-016d-472c-9c8b-b887d2278878"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:46:58.813347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.813312    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99hh\" (UniqueName: \"kubernetes.io/projected/8d45858c-016d-472c-9c8b-b887d2278878-kube-api-access-q99hh\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:58.813347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.813345    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:58.813347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.813356    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d45858c-016d-472c-9c8b-b887d2278878-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:58.835087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.835037    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj" podUID="c00342e8-823e-4555-a8a6-cd50bd08d790" containerName="payload-processing" containerID="cri-o://fffc0a65ab7863a7e2a4998a916cd3dee0da4954a1b4394f55942454e53547ae" gracePeriod=25
Nov 03 03:46:58.941444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:58.941412    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx" podUID="69ac2c61-e3c4-48f6-81c0-7caa4c894f91" containerName="payload-pre-processing" containerID="cri-o://cb63305a29e404c4aa0a1be5fb584c70db0ad097332a0086b1699210701f41eb" gracePeriod=25
Nov 03 03:46:59.068701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.068482    5515 generic.go:356] "Generic (PLEG): container finished" podID="8d45858c-016d-472c-9c8b-b887d2278878" containerID="01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779" exitCode=0
Nov 03 03:46:59.068701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.068566    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"
Nov 03 03:46:59.068701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.068636    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" event={"ID":"8d45858c-016d-472c-9c8b-b887d2278878","Type":"ContainerDied","Data":"01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779"}
Nov 03 03:46:59.068701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.068701    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp" event={"ID":"8d45858c-016d-472c-9c8b-b887d2278878","Type":"ContainerDied","Data":"b62813e99aed545b04253f11d58ba198e9ebd1914fd1cf9b52b33af419a80fe9"}
Nov 03 03:46:59.069003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.068732    5515 scope.go:122] "RemoveContainer" containerID="01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779"
Nov 03 03:46:59.072222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.072180    5515 generic.go:356] "Generic (PLEG): container finished" podID="69ac2c61-e3c4-48f6-81c0-7caa4c894f91" containerID="cb63305a29e404c4aa0a1be5fb584c70db0ad097332a0086b1699210701f41eb" exitCode=0
Nov 03 03:46:59.072347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.072259    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx" event={"ID":"69ac2c61-e3c4-48f6-81c0-7caa4c894f91","Type":"ContainerDied","Data":"cb63305a29e404c4aa0a1be5fb584c70db0ad097332a0086b1699210701f41eb"}
Nov 03 03:46:59.074277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.074252    5515 generic.go:356] "Generic (PLEG): container finished" podID="c00342e8-823e-4555-a8a6-cd50bd08d790" containerID="fffc0a65ab7863a7e2a4998a916cd3dee0da4954a1b4394f55942454e53547ae" exitCode=0
Nov 03 03:46:59.074382 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.074310    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj" event={"ID":"c00342e8-823e-4555-a8a6-cd50bd08d790","Type":"ContainerDied","Data":"fffc0a65ab7863a7e2a4998a916cd3dee0da4954a1b4394f55942454e53547ae"}
Nov 03 03:46:59.089040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.089017    5515 scope.go:122] "RemoveContainer" containerID="01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779"
Nov 03 03:46:59.089374 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:46:59.089341    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779\": container with ID starting with 01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779 not found: ID does not exist" containerID="01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779"
Nov 03 03:46:59.089464 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.089388    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779"} err="failed to get container status \"01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779\": rpc error: code = NotFound desc = could not find container \"01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779\": container with ID starting with 01592115d7eaf01b56216187ad0921db48dbf6bf714825d99077802772113779 not found: ID does not exist"
Nov 03 03:46:59.111508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.111482    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:46:59.199670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.199649    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:46:59.220226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.220184    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn\" (UniqueName: \"kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn\") pod \"c00342e8-823e-4555-a8a6-cd50bd08d790\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") "
Nov 03 03:46:59.220362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.220264    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume\") pod \"c00342e8-823e-4555-a8a6-cd50bd08d790\" (UID: \"c00342e8-823e-4555-a8a6-cd50bd08d790\") "
Nov 03 03:46:59.220659 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.220636    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume" pod "c00342e8-823e-4555-a8a6-cd50bd08d790" (UID: "c00342e8-823e-4555-a8a6-cd50bd08d790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:59.220752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.220730    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00342e8-823e-4555-a8a6-cd50bd08d790-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:59.222488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.222464    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn" pod "c00342e8-823e-4555-a8a6-cd50bd08d790" (UID: "c00342e8-823e-4555-a8a6-cd50bd08d790"). InnerVolumeSpecName "kube-api-access-vz6pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:59.235789 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.235765    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"]
Nov 03 03:46:59.239426 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.239408    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3480a2-7554786668-4q8cp"]
Nov 03 03:46:59.322160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.322122    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume\") pod \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") "
Nov 03 03:46:59.322343 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.322190    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx\" (UniqueName: \"kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx\") pod \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\" (UID: \"69ac2c61-e3c4-48f6-81c0-7caa4c894f91\") "
Nov 03 03:46:59.322525 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.322504    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume" pod "69ac2c61-e3c4-48f6-81c0-7caa4c894f91" (UID: "69ac2c61-e3c4-48f6-81c0-7caa4c894f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:46:59.322631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.322585    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz6pn\" (UniqueName: \"kubernetes.io/projected/c00342e8-823e-4555-a8a6-cd50bd08d790-kube-api-access-vz6pn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:59.324409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.324391    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx" pod "69ac2c61-e3c4-48f6-81c0-7caa4c894f91" (UID: "69ac2c61-e3c4-48f6-81c0-7caa4c894f91"). InnerVolumeSpecName "kube-api-access-px8lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:46:59.423575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.423521    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:46:59.423966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:46:59.423583    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8lx\" (UniqueName: \"kubernetes.io/projected/69ac2c61-e3c4-48f6-81c0-7caa4c894f91-kube-api-access-px8lx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:47:00.081738 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.081708    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"
Nov 03 03:47:00.081931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.081704    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx" event={"ID":"69ac2c61-e3c4-48f6-81c0-7caa4c894f91","Type":"ContainerDied","Data":"4b22bc7aba541b8c8f8e3ce0a9c09e414979b0960b68cc5b96b716b7a113d428"}
Nov 03 03:47:00.081931 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.081834    5515 scope.go:122] "RemoveContainer" containerID="cb63305a29e404c4aa0a1be5fb584c70db0ad097332a0086b1699210701f41eb"
Nov 03 03:47:00.083195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.083138    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"
Nov 03 03:47:00.083195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.083169    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj" event={"ID":"c00342e8-823e-4555-a8a6-cd50bd08d790","Type":"ContainerDied","Data":"2a3c3cd2607d5eae5645f45b0e04b932458beee06ce4e831112234eb46291ef0"}
Nov 03 03:47:00.092176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.091893    5515 scope.go:122] "RemoveContainer" containerID="fffc0a65ab7863a7e2a4998a916cd3dee0da4954a1b4394f55942454e53547ae"
Nov 03 03:47:00.249188 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.249154    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"]
Nov 03 03:47:00.299123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.299090    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a9e5cd-57f6cdccf5-z44rx"]
Nov 03 03:47:00.364989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.364946    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"]
Nov 03 03:47:00.372430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.372405    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a9e5cd-6cb76d7cd7-np4lj"]
Nov 03 03:47:00.428882 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.428847    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:00.628839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.628754    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:00.894978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.894909    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ac2c61-e3c4-48f6-81c0-7caa4c894f91" path="/var/lib/kubelet/pods/69ac2c61-e3c4-48f6-81c0-7caa4c894f91/volumes"
Nov 03 03:47:00.895339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.895325    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d45858c-016d-472c-9c8b-b887d2278878" path="/var/lib/kubelet/pods/8d45858c-016d-472c-9c8b-b887d2278878/volumes"
Nov 03 03:47:00.895734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:00.895719    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00342e8-823e-4555-a8a6-cd50bd08d790" path="/var/lib/kubelet/pods/c00342e8-823e-4555-a8a6-cd50bd08d790/volumes"
Nov 03 03:47:01.534539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:01.534508    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:01.677704 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:01.677674    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:03.111688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:03.111655    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:03.259895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:03.259859    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:03.666123 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:03.666096    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:05.735243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:05.735212    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:10.951214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:10.951182    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:11.113217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:11.113183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:11.258277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:11.258181    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:16.432099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:16.432068    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:19.619635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:19.619600    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:47:20.984269 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:47:20.984226    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:47:21.427541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:21.427486    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:22.546938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:22.546908    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:22.687213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:22.687182    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:26.255362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:26.255330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:30.893311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893279    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"]
Nov 03 03:47:30.893772 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893756    5515 cpu_manager.go:414] "containerMap: removing container" podUID="69ac2c61-e3c4-48f6-81c0-7caa4c894f91" containerName="payload-pre-processing"
Nov 03 03:47:30.893827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893773    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ac2c61-e3c4-48f6-81c0-7caa4c894f91" containerName="payload-pre-processing"
Nov 03 03:47:30.893827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893787    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c00342e8-823e-4555-a8a6-cd50bd08d790" containerName="payload-processing"
Nov 03 03:47:30.893827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893792    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00342e8-823e-4555-a8a6-cd50bd08d790" containerName="payload-processing"
Nov 03 03:47:30.893827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893801    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8d45858c-016d-472c-9c8b-b887d2278878" containerName="maas-api"
Nov 03 03:47:30.893827 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893805    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d45858c-016d-472c-9c8b-b887d2278878" containerName="maas-api"
Nov 03 03:47:30.893989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893910    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ac2c61-e3c4-48f6-81c0-7caa4c894f91" containerName="payload-pre-processing"
Nov 03 03:47:30.893989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893929    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00342e8-823e-4555-a8a6-cd50bd08d790" containerName="payload-processing"
Nov 03 03:47:30.893989 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.893936    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d45858c-016d-472c-9c8b-b887d2278878" containerName="maas-api"
Nov 03 03:47:30.897339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.897323    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:30.951982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.951953    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"]
Nov 03 03:47:30.978124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.978093    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:30.978302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.978282    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pkn\" (UniqueName: \"kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:30.978350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.978318    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:30.978350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.978342    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:30.999845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:30.999815    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e84ee2-serving-cert\""
Nov 03 03:47:31.080064 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.080033    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.080231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.080168    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pkn\" (UniqueName: \"kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.080231 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:47:31.080171    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-serving-cert: secret "maas-api-e2e-praxis-b-e84ee2-serving-cert" not found
Nov 03 03:47:31.080231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.080193    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.080231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.080216    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.080430 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:47:31.080242    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls podName:e18ba05a-3609-4759-847b-6cc1a4a22d77 nodeName:}" failed. No retries permitted until 2026-11-03 03:47:31.580220269 +0000 UTC m=+2742.811112665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls") pod "maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77") : secret "maas-api-e2e-praxis-b-e84ee2-serving-cert" not found
Nov 03 03:47:31.080978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.080894    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.082694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.082674    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.085486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.085459    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:31.149345 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.148642    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pkn\" (UniqueName: \"kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.249106 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.249072    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:31.391910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.391872    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:31.587242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.587148    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.590019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.589981    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:31.810675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:31.810643    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:32.189769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:32.189740    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"]
Nov 03 03:47:32.190851 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:47:32.190817    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18ba05a_3609_4759_847b_6cc1a4a22d77.slice/crio-d9e8ac163a422584981d6ec6f753906715a97ffcf2cd86e76715cc7426968a1c WatchSource:0}: Error finding container d9e8ac163a422584981d6ec6f753906715a97ffcf2cd86e76715cc7426968a1c: Status 404 returned error can't find the container with id d9e8ac163a422584981d6ec6f753906715a97ffcf2cd86e76715cc7426968a1c
Nov 03 03:47:32.282631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:32.282594    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" event={"ID":"e18ba05a-3609-4759-847b-6cc1a4a22d77","Type":"ContainerStarted","Data":"d9e8ac163a422584981d6ec6f753906715a97ffcf2cd86e76715cc7426968a1c"}
Nov 03 03:47:33.290314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:33.290276    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" event={"ID":"e18ba05a-3609-4759-847b-6cc1a4a22d77","Type":"ContainerStarted","Data":"7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599"}
Nov 03 03:47:33.290760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:33.290409    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:33.411336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:33.411283    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" podStartSLOduration=2.542268605 podStartE2EDuration="3.411265523s" totalImagesPullingTime="868.996918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:47:33.409975514 +0000 UTC m=+2744.640867934" watchObservedRunningTime="2026-11-03 03:47:33.411265523 +0000 UTC m=+2744.642157942"
Nov 03 03:47:36.137793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:36.137760    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:39.301702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:39.301673    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:47:41.040759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:41.040721    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:45.927358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:45.927320    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:50.208292 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:50.208245    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:50.984944 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:47:50.984863    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:47:52.076324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:52.076289    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:47:56.926865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:47:56.926822    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:00.655949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:00.655891    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:00.801794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:00.801752    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:01.873894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:01.873854    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:02.350263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:02.350172    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:02.835369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:02.835337    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:03.602122 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:03.602086    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:07.156834 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:07.156799    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:10.339320 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:10.339285    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:48:12.027587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:12.027534    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:14.246928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:14.246893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:14.896847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:14.896816    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"]
Nov 03 03:48:14.900894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:14.900878    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:14.954411 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:14.954383    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-de27f5-openshift-default-dockercfg-djfq7\""
Nov 03 03:48:15.008511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.008478    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"]
Nov 03 03:48:15.015030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015000    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015033    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015052    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015069    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015241    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gzpn\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015291    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015403    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015461    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.015592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.015575    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116205    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116275    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116303    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116339    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116359    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116381    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116406    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116450    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gzpn\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116480    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116832    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.116923 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116874    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.117008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.116983    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.117111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.117022    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.117242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.117224    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.118906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.118876    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.119182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.119163    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.178544 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.178468    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.178774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.178754    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gzpn\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.214978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.214949    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:15.474675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.474596    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:15.643898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.643870    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"]
Nov 03 03:48:15.645289 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:48:15.645262    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc82122b_3adb_4448_897a_4165d1c711b1.slice/crio-b1d231e08b5aee7322dc37d7aecb550807c778bde72e0559c7ba92a7fb643a52 WatchSource:0}: Error finding container b1d231e08b5aee7322dc37d7aecb550807c778bde72e0559c7ba92a7fb643a52: Status 404 returned error can't find the container with id b1d231e08b5aee7322dc37d7aecb550807c778bde72e0559c7ba92a7fb643a52
Nov 03 03:48:15.647677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.647645    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:48:15.647766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.647708    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:48:15.647766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:15.647736    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:48:16.364999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:16.364967    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:16.549616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:16.549582    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" event={"ID":"cc82122b-3adb-4448-897a-4165d1c711b1","Type":"ContainerStarted","Data":"c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101"}
Nov 03 03:48:16.550058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:16.549635    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" event={"ID":"cc82122b-3adb-4448-897a-4165d1c711b1","Type":"ContainerStarted","Data":"b1d231e08b5aee7322dc37d7aecb550807c778bde72e0559c7ba92a7fb643a52"}
Nov 03 03:48:16.581576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:16.580575    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:17.133134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:17.133088    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:17.215865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:17.215835    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:17.217317 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:17.217292    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 03:48:17.217451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:17.217339    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 03:48:18.215278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:18.215239    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 03:48:18.215694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:18.215308    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 03:48:19.216252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:19.216219    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 03:48:19.216721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:19.216284    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 03:48:20.216060 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:20.216024    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 03:48:20.216241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:20.216080    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 03:48:20.839055 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:20.839020    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:21.007719 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:48:21.007677    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:48:21.013414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:21.013385    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:21.184735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:21.184695    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:21.215980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:21.215947    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 03:48:21.216186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:21.216008    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 03:48:21.512328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:21.512232    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:21.656711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:21.656671    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:22.215762 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:22.215724    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 03 03:48:22.216211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:22.215794    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 03 03:48:22.234657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:22.234619    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:23.392761 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:23.392724    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:23.393576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:23.393510    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:23.394340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:23.394308    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 03:48:23.617587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:23.617524    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podStartSLOduration=9.617509701 podStartE2EDuration="9.617509701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:16.687640991 +0000 UTC m=+2787.918533414" watchObservedRunningTime="2026-11-03 03:48:23.617509701 +0000 UTC m=+2794.848402119"
Nov 03 03:48:24.125795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:24.125757    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:24.300941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:24.300908    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:27.477268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:27.477233    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:32.332776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:32.332742    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:34.506818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:34.506787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:37.527231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:37.527195    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:37.848219 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:37.848135    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:39.601111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:39.601074    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:39.740575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:39.740518    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:42.428145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:42.428091    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:43.642790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:43.642753    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"]
Nov 03 03:48:43.751996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:43.751963    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"]
Nov 03 03:48:44.000101 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:44.000023    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:44.142253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:44.142217    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:44.403412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:44.403378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:47.714486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:47.714452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:48.643852 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:48.643810    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk" podUID="68a9e5f0-b6aa-44fb-b643-05b2ea04a070" containerName="payload-processing" containerID="cri-o://90dc797f166100f0a181651d7176c7136741ce52d627499cc030d42bfb2439e9" gracePeriod=25
Nov 03 03:48:48.752971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:48.752939    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr" podUID="e21bad23-57d9-483e-ac2d-dabcdc8f9f41" containerName="payload-pre-processing" containerID="cri-o://92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d" gracePeriod=25
Nov 03 03:48:48.787423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:48.787395    5515 generic.go:356] "Generic (PLEG): container finished" podID="68a9e5f0-b6aa-44fb-b643-05b2ea04a070" containerID="90dc797f166100f0a181651d7176c7136741ce52d627499cc030d42bfb2439e9" exitCode=0
Nov 03 03:48:48.787595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:48.787468    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk" event={"ID":"68a9e5f0-b6aa-44fb-b643-05b2ea04a070","Type":"ContainerDied","Data":"90dc797f166100f0a181651d7176c7136741ce52d627499cc030d42bfb2439e9"}
Nov 03 03:48:48.918855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:48.918834    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:48:49.025361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.025339    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:48:49.041823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.041798    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume\") pod \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") "
Nov 03 03:48:49.041970 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.041865    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj\" (UniqueName: \"kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj\") pod \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\" (UID: \"68a9e5f0-b6aa-44fb-b643-05b2ea04a070\") "
Nov 03 03:48:49.042263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.042234    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume" pod "68a9e5f0-b6aa-44fb-b643-05b2ea04a070" (UID: "68a9e5f0-b6aa-44fb-b643-05b2ea04a070"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:48:49.042602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.042581    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:48:49.044618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.044593    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj" pod "68a9e5f0-b6aa-44fb-b643-05b2ea04a070" (UID: "68a9e5f0-b6aa-44fb-b643-05b2ea04a070"). InnerVolumeSpecName "kube-api-access-hjxsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:48:49.144042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.144019    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp\" (UniqueName: \"kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp\") pod \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") "
Nov 03 03:48:49.144214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.144092    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume\") pod \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\" (UID: \"e21bad23-57d9-483e-ac2d-dabcdc8f9f41\") "
Nov 03 03:48:49.144342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.144328    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjxsj\" (UniqueName: \"kubernetes.io/projected/68a9e5f0-b6aa-44fb-b643-05b2ea04a070-kube-api-access-hjxsj\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:48:49.144471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.144449    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume" pod "e21bad23-57d9-483e-ac2d-dabcdc8f9f41" (UID: "e21bad23-57d9-483e-ac2d-dabcdc8f9f41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:48:49.146114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.146097    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp" pod "e21bad23-57d9-483e-ac2d-dabcdc8f9f41" (UID: "e21bad23-57d9-483e-ac2d-dabcdc8f9f41"). InnerVolumeSpecName "kube-api-access-jlltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:48:49.245487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.245459    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlltp\" (UniqueName: \"kubernetes.io/projected/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-kube-api-access-jlltp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:48:49.245487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.245485    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e21bad23-57d9-483e-ac2d-dabcdc8f9f41-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:48:49.795594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.795509    5515 generic.go:356] "Generic (PLEG): container finished" podID="e21bad23-57d9-483e-ac2d-dabcdc8f9f41" containerID="92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d" exitCode=0
Nov 03 03:48:49.795994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.795565    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"
Nov 03 03:48:49.795994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.795627    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr" event={"ID":"e21bad23-57d9-483e-ac2d-dabcdc8f9f41","Type":"ContainerDied","Data":"92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d"}
Nov 03 03:48:49.795994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.795697    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr" event={"ID":"e21bad23-57d9-483e-ac2d-dabcdc8f9f41","Type":"ContainerDied","Data":"31db461b73b18e002bc426993cbe885415c4744d3583f6745a08f7d7cc0f3f3a"}
Nov 03 03:48:49.795994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.795722    5515 scope.go:122] "RemoveContainer" containerID="92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d"
Nov 03 03:48:49.797140 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.797121    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"
Nov 03 03:48:49.797206 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.797151    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk" event={"ID":"68a9e5f0-b6aa-44fb-b643-05b2ea04a070","Type":"ContainerDied","Data":"34e5c094dd9531891d40518ba8e6d7f5cbb0644d396f5c75af7e9cfd78fa6037"}
Nov 03 03:48:49.810271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.810255    5515 scope.go:122] "RemoveContainer" containerID="92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d"
Nov 03 03:48:49.810575 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:48:49.810531    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d\": container with ID starting with 92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d not found: ID does not exist" containerID="92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d"
Nov 03 03:48:49.810627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.810583    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d"} err="failed to get container status \"92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d\": rpc error: code = NotFound desc = could not find container \"92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d\": container with ID starting with 92e7d1fb95b7d8d304d28631d72a1bc467c1d556814968c21871024c46084e5d not found: ID does not exist"
Nov 03 03:48:49.810627 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.810598    5515 scope.go:122] "RemoveContainer" containerID="90dc797f166100f0a181651d7176c7136741ce52d627499cc030d42bfb2439e9"
Nov 03 03:48:49.963794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:49.963720    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"]
Nov 03 03:48:50.014847 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:50.014818    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2be7ec-69d9fbdbf5-zwlsk"]
Nov 03 03:48:50.078190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:50.078157    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"]
Nov 03 03:48:50.081806 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:50.081778    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2be7ec-57d85b6459-l6sfr"]
Nov 03 03:48:50.458494 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:50.458462    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:50.894168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:50.894136    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a9e5f0-b6aa-44fb-b643-05b2ea04a070" path="/var/lib/kubelet/pods/68a9e5f0-b6aa-44fb-b643-05b2ea04a070/volumes"
Nov 03 03:48:50.894578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:50.894562    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21bad23-57d9-483e-ac2d-dabcdc8f9f41" path="/var/lib/kubelet/pods/e21bad23-57d9-483e-ac2d-dabcdc8f9f41/volumes"
Nov 03 03:48:50.985570 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:48:50.985468    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:48:52.634581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.634517    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:52.944643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.944572    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"]
Nov 03 03:48:52.945087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.945074    5515 cpu_manager.go:414] "containerMap: removing container" podUID="68a9e5f0-b6aa-44fb-b643-05b2ea04a070" containerName="payload-processing"
Nov 03 03:48:52.945137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.945088    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a9e5f0-b6aa-44fb-b643-05b2ea04a070" containerName="payload-processing"
Nov 03 03:48:52.945137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.945114    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e21bad23-57d9-483e-ac2d-dabcdc8f9f41" containerName="payload-pre-processing"
Nov 03 03:48:52.945137 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.945120    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21bad23-57d9-483e-ac2d-dabcdc8f9f41" containerName="payload-pre-processing"
Nov 03 03:48:52.945241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.945200    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21bad23-57d9-483e-ac2d-dabcdc8f9f41" containerName="payload-pre-processing"
Nov 03 03:48:52.945241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.945216    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a9e5f0-b6aa-44fb-b643-05b2ea04a070" containerName="payload-processing"
Nov 03 03:48:52.950077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:52.950060    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.003425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.003407    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b1ad4e-openshift-default-dockercfg-m64tn\""
Nov 03 03:48:53.056697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.056674    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"]
Nov 03 03:48:53.089985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.089951    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.089990    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090019    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090100    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090215    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090268    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090372    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090482 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090453    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.090631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.090534    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192043    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192115    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192201    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192244    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192290    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192339    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192376    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192416    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192528 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192448    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192685    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192771    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.192804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192788    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.193222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.192841    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.193222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.193128    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.194759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.194706    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.194947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.194930    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.254091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.254064    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.254241 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.254222    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.263114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.263092    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:53.689566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.689528    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"]
Nov 03 03:48:53.691446 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:48:53.691421    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8cc06c_dc7b_4005_a69a_022fac0d2642.slice/crio-9440e8950b0c78f49ad8f918cd5bd9e4c9468353e2f16813db6ec8cf77c55c6b WatchSource:0}: Error finding container 9440e8950b0c78f49ad8f918cd5bd9e4c9468353e2f16813db6ec8cf77c55c6b: Status 404 returned error can't find the container with id 9440e8950b0c78f49ad8f918cd5bd9e4c9468353e2f16813db6ec8cf77c55c6b
Nov 03 03:48:53.694024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.693992    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:48:53.694108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.694054    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:48:53.694108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.694082    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:48:53.821315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.821277    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" event={"ID":"4b8cc06c-dc7b-4005-a69a-022fac0d2642","Type":"ContainerStarted","Data":"cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735"}
Nov 03 03:48:53.821495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.821334    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" event={"ID":"4b8cc06c-dc7b-4005-a69a-022fac0d2642","Type":"ContainerStarted","Data":"9440e8950b0c78f49ad8f918cd5bd9e4c9468353e2f16813db6ec8cf77c55c6b"}
Nov 03 03:48:53.944456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:53.944352    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podStartSLOduration=1.94433504 podStartE2EDuration="1.94433504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:48:53.942845223 +0000 UTC m=+2825.173737642" watchObservedRunningTime="2026-11-03 03:48:53.94433504 +0000 UTC m=+2825.175227459"
Nov 03 03:48:54.265598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:54.265469    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:48:54.265879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:54.265839    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:48:54.266302 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:54.266273    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:48:55.266574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:55.266059    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:48:55.266574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:55.266124    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:48:56.288577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:56.280879    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:48:56.288577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:56.280942    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:48:57.286576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:57.282927    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:48:57.286576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:57.282997    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:48:57.657564 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:57.655848    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:48:58.286576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:58.285656    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:48:58.286576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:58.285726    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:48:59.274577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:59.266916    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:48:59.274577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:48:59.267016    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:49:00.263753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:00.263720    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 03 03:49:00.263948 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:00.263778    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 03 03:49:02.228247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:02.228213    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:49:02.228699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:02.228511    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:49:02.229252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:02.229234    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:49:02.550301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:02.550224    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:03.957514 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:03.957485    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:06.661004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.660971    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"]
Nov 03 03:49:06.661413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.661176    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" podUID="f5db33db-363e-4986-ac5c-97875562eb05" containerName="maas-api" containerID="cri-o://467d7216184f0cd5a33543e7394a433b1671637d6fbd9a1838894eea1cf25687" gracePeriod=30
Nov 03 03:49:06.764720 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.764675    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:06.915792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.915753    5515 generic.go:356] "Generic (PLEG): container finished" podID="f5db33db-363e-4986-ac5c-97875562eb05" containerID="467d7216184f0cd5a33543e7394a433b1671637d6fbd9a1838894eea1cf25687" exitCode=0
Nov 03 03:49:06.915959 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.915905    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" event={"ID":"f5db33db-363e-4986-ac5c-97875562eb05","Type":"ContainerDied","Data":"467d7216184f0cd5a33543e7394a433b1671637d6fbd9a1838894eea1cf25687"}
Nov 03 03:49:06.940405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.940383    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:49:06.996385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.996363    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t\" (UniqueName: \"kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t\") pod \"f5db33db-363e-4986-ac5c-97875562eb05\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") "
Nov 03 03:49:06.996513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.996394    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs\") pod \"f5db33db-363e-4986-ac5c-97875562eb05\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") "
Nov 03 03:49:06.996513 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.996473    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\") pod \"f5db33db-363e-4986-ac5c-97875562eb05\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") "
Nov 03 03:49:06.996677 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.996653    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca\") pod \"f5db33db-363e-4986-ac5c-97875562eb05\" (UID: \"f5db33db-363e-4986-ac5c-97875562eb05\") "
Nov 03 03:49:06.996992 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.996963    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca" pod "f5db33db-363e-4986-ac5c-97875562eb05" (UID: "f5db33db-363e-4986-ac5c-97875562eb05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:49:06.997341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.997319    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5db33db-363e-4986-ac5c-97875562eb05-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:06.998769 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.998741    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t" pod "f5db33db-363e-4986-ac5c-97875562eb05" (UID: "f5db33db-363e-4986-ac5c-97875562eb05"). InnerVolumeSpecName "kube-api-access-gss4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:49:06.998863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.998790    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs" pod "f5db33db-363e-4986-ac5c-97875562eb05" (UID: "f5db33db-363e-4986-ac5c-97875562eb05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:49:06.998863 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:06.998827    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls" pod "f5db33db-363e-4986-ac5c-97875562eb05" (UID: "f5db33db-363e-4986-ac5c-97875562eb05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:49:07.098133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:07.098100    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gss4t\" (UniqueName: \"kubernetes.io/projected/f5db33db-363e-4986-ac5c-97875562eb05-kube-api-access-gss4t\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:07.098133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:07.098128    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:07.098133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:07.098137    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5db33db-363e-4986-ac5c-97875562eb05-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:07.922210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:07.922178    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq" event={"ID":"f5db33db-363e-4986-ac5c-97875562eb05","Type":"ContainerDied","Data":"029395295fef6b4c7b5a527c87b8f5ec855654bde2363d709e7003ab445a03b3"}
Nov 03 03:49:07.922210 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:07.922197    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"
Nov 03 03:49:07.922782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:07.922239    5515 scope.go:122] "RemoveContainer" containerID="467d7216184f0cd5a33543e7394a433b1671637d6fbd9a1838894eea1cf25687"
Nov 03 03:49:08.089596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:08.089564    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"]
Nov 03 03:49:08.140008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:08.139971    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2be7ec-7d8498c847-mv2xq"]
Nov 03 03:49:08.528186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:08.528153    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:08.893539 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:08.893507    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5db33db-363e-4986-ac5c-97875562eb05" path="/var/lib/kubelet/pods/f5db33db-363e-4986-ac5c-97875562eb05/volumes"
Nov 03 03:49:09.961283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:09.961250    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:49:13.086300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.086270    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"]
Nov 03 03:49:13.086848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.086833    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f5db33db-363e-4986-ac5c-97875562eb05" containerName="maas-api"
Nov 03 03:49:13.086890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.086850    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5db33db-363e-4986-ac5c-97875562eb05" containerName="maas-api"
Nov 03 03:49:13.086973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.086962    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5db33db-363e-4986-ac5c-97875562eb05" containerName="maas-api"
Nov 03 03:49:13.090293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.090278    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.146839 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.146805    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-8950af-serving-cert\""
Nov 03 03:49:13.171588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.171529    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94bx\" (UniqueName: \"kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.171746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.171708    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.171746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.171731    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.171838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.171756    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.202369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.202334    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"]
Nov 03 03:49:13.272924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.272893    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.272924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.272933    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.273209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.272966    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.273209 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:49:13.273075    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-serving-cert: secret "maas-api-e2e-praxis-b-8950af-serving-cert" not found
Nov 03 03:49:13.273209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.273096    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94bx\" (UniqueName: \"kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.273209 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:49:13.273147    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls podName:a83c7e53-341c-42d8-864e-6c373311755e nodeName:}" failed. No retries permitted until 2026-11-03 03:49:13.773125399 +0000 UTC m=+2845.004017801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls") pod "maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" (UID: "a83c7e53-341c-42d8-864e-6c373311755e") : secret "maas-api-e2e-praxis-b-8950af-serving-cert" not found
Nov 03 03:49:13.273691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.273674    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.275517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.275496    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.289616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.289572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:13.361793 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.361720    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94bx\" (UniqueName: \"kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.446455 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.446416    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:13.596529 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.596499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:13.779538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.779451    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:13.782084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:13.782060    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8950af-8656794869-f8v6b\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:14.004309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:14.004278    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:14.378019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:14.377994    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"]
Nov 03 03:49:14.379627 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:49:14.379602    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c7e53_341c_42d8_864e_6c373311755e.slice/crio-d9a63e70f754577a95dc92c86d9881d65286ac3b90058a12ff255070d543ae35 WatchSource:0}: Error finding container d9a63e70f754577a95dc92c86d9881d65286ac3b90058a12ff255070d543ae35: Status 404 returned error can't find the container with id d9a63e70f754577a95dc92c86d9881d65286ac3b90058a12ff255070d543ae35
Nov 03 03:49:14.976239 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:14.976204    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" event={"ID":"a83c7e53-341c-42d8-864e-6c373311755e","Type":"ContainerStarted","Data":"d9a63e70f754577a95dc92c86d9881d65286ac3b90058a12ff255070d543ae35"}
Nov 03 03:49:15.023726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:15.023666    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:15.170801 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:15.170763    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:15.763708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:15.763680    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:15.982940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:15.982858    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" event={"ID":"a83c7e53-341c-42d8-864e-6c373311755e","Type":"ContainerStarted","Data":"e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd"}
Nov 03 03:49:15.983091 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:15.982937    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:16.103541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:16.103490    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" podStartSLOduration=1.823328082 podStartE2EDuration="3.103475137s" totalImagesPullingTime="1.280147055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:16.102264891 +0000 UTC m=+2847.333157310" watchObservedRunningTime="2026-11-03 03:49:16.103475137 +0000 UTC m=+2847.334367574"
Nov 03 03:49:16.695480 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:16.695403    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:17.237870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:17.237836    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:18.020694 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:18.020661    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:18.731518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:18.731488    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:20.800067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:20.800038    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:20.984231 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:49:20.984194    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:49:21.994252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:21.994219    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:49:23.869684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:23.869649    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:24.325841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:24.325759    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:24.824197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:24.824166    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"]
Nov 03 03:49:24.936234 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:24.936201    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"]
Nov 03 03:49:25.185248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:25.185216    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:25.336832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:25.336787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:28.941751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:28.941717    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:29.825367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:29.825322    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz" podUID="58c1f323-95e3-47ff-90f7-b41f093862fe" containerName="payload-processing" containerID="cri-o://10a56bbf2d6cf772d7ec36d8d81c23ca65e09435afb06bdd64d414edb13484df" gracePeriod=25
Nov 03 03:49:29.936695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:29.936663    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc" podUID="f96cc613-0061-4097-8491-29ed21891d63" containerName="payload-pre-processing" containerID="cri-o://0c1778b17e33f31011b541207463b4c77b52c5a4852e7f88b8ffbc064237edfd" gracePeriod=25
Nov 03 03:49:30.062512 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.062462    5515 generic.go:356] "Generic (PLEG): container finished" podID="58c1f323-95e3-47ff-90f7-b41f093862fe" containerID="10a56bbf2d6cf772d7ec36d8d81c23ca65e09435afb06bdd64d414edb13484df" exitCode=0
Nov 03 03:49:30.062892 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.062532    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz" event={"ID":"58c1f323-95e3-47ff-90f7-b41f093862fe","Type":"ContainerDied","Data":"10a56bbf2d6cf772d7ec36d8d81c23ca65e09435afb06bdd64d414edb13484df"}
Nov 03 03:49:30.064617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.064593    5515 generic.go:356] "Generic (PLEG): container finished" podID="f96cc613-0061-4097-8491-29ed21891d63" containerID="0c1778b17e33f31011b541207463b4c77b52c5a4852e7f88b8ffbc064237edfd" exitCode=0
Nov 03 03:49:30.064731 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.064635    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc" event={"ID":"f96cc613-0061-4097-8491-29ed21891d63","Type":"ContainerDied","Data":"0c1778b17e33f31011b541207463b4c77b52c5a4852e7f88b8ffbc064237edfd"}
Nov 03 03:49:30.097058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.097035    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:49:30.116900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.116821    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:30.197817 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.197794    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:49:30.200925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.200906    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2\" (UniqueName: \"kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2\") pod \"58c1f323-95e3-47ff-90f7-b41f093862fe\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") "
Nov 03 03:49:30.201136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.201117    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume\") pod \"58c1f323-95e3-47ff-90f7-b41f093862fe\" (UID: \"58c1f323-95e3-47ff-90f7-b41f093862fe\") "
Nov 03 03:49:30.201443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.201423    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume" pod "58c1f323-95e3-47ff-90f7-b41f093862fe" (UID: "58c1f323-95e3-47ff-90f7-b41f093862fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:49:30.201739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.201560    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58c1f323-95e3-47ff-90f7-b41f093862fe-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:30.203138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.203115    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2" pod "58c1f323-95e3-47ff-90f7-b41f093862fe" (UID: "58c1f323-95e3-47ff-90f7-b41f093862fe"). InnerVolumeSpecName "kube-api-access-xwxv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:49:30.302428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.302401    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z\" (UniqueName: \"kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z\") pod \"f96cc613-0061-4097-8491-29ed21891d63\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") "
Nov 03 03:49:30.302596 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.302576    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume\") pod \"f96cc613-0061-4097-8491-29ed21891d63\" (UID: \"f96cc613-0061-4097-8491-29ed21891d63\") "
Nov 03 03:49:30.302906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.302877    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume" pod "f96cc613-0061-4097-8491-29ed21891d63" (UID: "f96cc613-0061-4097-8491-29ed21891d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:49:30.302906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.302893    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxv2\" (UniqueName: \"kubernetes.io/projected/58c1f323-95e3-47ff-90f7-b41f093862fe-kube-api-access-xwxv2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:30.304654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.304631    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z" pod "f96cc613-0061-4097-8491-29ed21891d63" (UID: "f96cc613-0061-4097-8491-29ed21891d63"). InnerVolumeSpecName "kube-api-access-bjx8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:49:30.403967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.403937    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96cc613-0061-4097-8491-29ed21891d63-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:30.403967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:30.403963    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjx8z\" (UniqueName: \"kubernetes.io/projected/f96cc613-0061-4097-8491-29ed21891d63-kube-api-access-bjx8z\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:49:31.071771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.071740    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"
Nov 03 03:49:31.072195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.071744    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz" event={"ID":"58c1f323-95e3-47ff-90f7-b41f093862fe","Type":"ContainerDied","Data":"16a80eea6bd9cb1c849f66a7e7f8ae1f36cc12a950ea11497de2e4028ed93796"}
Nov 03 03:49:31.072195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.071906    5515 scope.go:122] "RemoveContainer" containerID="10a56bbf2d6cf772d7ec36d8d81c23ca65e09435afb06bdd64d414edb13484df"
Nov 03 03:49:31.073278 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.073256    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc" event={"ID":"f96cc613-0061-4097-8491-29ed21891d63","Type":"ContainerDied","Data":"31b69929097ae9daa8e576b5ab12dc2c01cabd37c2611ae5bce16beb073a66ba"}
Nov 03 03:49:31.073342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.073288    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"
Nov 03 03:49:31.083035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.083017    5515 scope.go:122] "RemoveContainer" containerID="0c1778b17e33f31011b541207463b4c77b52c5a4852e7f88b8ffbc064237edfd"
Nov 03 03:49:31.243324 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.243293    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"]
Nov 03 03:49:31.246685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.246663    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e84ee2-b959d8559-ss6cz"]
Nov 03 03:49:31.357365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.357337    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"]
Nov 03 03:49:31.362246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:31.362224    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e84ee2-6448cd58c9-glsjc"]
Nov 03 03:49:32.894000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:32.893968    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c1f323-95e3-47ff-90f7-b41f093862fe" path="/var/lib/kubelet/pods/58c1f323-95e3-47ff-90f7-b41f093862fe/volumes"
Nov 03 03:49:32.894439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:32.894378    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96cc613-0061-4097-8491-29ed21891d63" path="/var/lib/kubelet/pods/f96cc613-0061-4097-8491-29ed21891d63/volumes"
Nov 03 03:49:33.830346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:33.830314    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:37.963398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.963364    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"]
Nov 03 03:49:37.964174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.964152    5515 cpu_manager.go:414] "containerMap: removing container" podUID="58c1f323-95e3-47ff-90f7-b41f093862fe" containerName="payload-processing"
Nov 03 03:49:37.964226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.964180    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c1f323-95e3-47ff-90f7-b41f093862fe" containerName="payload-processing"
Nov 03 03:49:37.964263 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.964255    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f96cc613-0061-4097-8491-29ed21891d63" containerName="payload-pre-processing"
Nov 03 03:49:37.964303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.964265    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96cc613-0061-4097-8491-29ed21891d63" containerName="payload-pre-processing"
Nov 03 03:49:37.964437 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.964423    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96cc613-0061-4097-8491-29ed21891d63" containerName="payload-pre-processing"
Nov 03 03:49:37.964472 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.964450    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c1f323-95e3-47ff-90f7-b41f093862fe" containerName="payload-processing"
Nov 03 03:49:37.978947 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:37.978919    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.087462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.087426    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"]
Nov 03 03:49:38.094579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.094514    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.095039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.095013    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.095205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.095188    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.095392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.095375    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkwlp\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.095530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.095512    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.095701 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.095671    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.097374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.097350    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.097604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.097585    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.097753 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.097735    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199139    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199199    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199240    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199282    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199315    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199426    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199486    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199522    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199627    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkwlp\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199665    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.199938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199924    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.200009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.199936    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.200404 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.200217    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.200656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.200632    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.203182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.203154    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.203478 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.203454    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.261424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.261368    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.261785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.261732    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkwlp\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.296994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.296969    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:38.736051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.736020    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"]
Nov 03 03:49:38.737392 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:49:38.737358    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f5990c_30ac_495d_851a_a865a50e5286.slice/crio-af8b41fafd8eb3a3869281cf6a4c5800ef9c0cedf14605c55366158fdf9c95df WatchSource:0}: Error finding container af8b41fafd8eb3a3869281cf6a4c5800ef9c0cedf14605c55366158fdf9c95df: Status 404 returned error can't find the container with id af8b41fafd8eb3a3869281cf6a4c5800ef9c0cedf14605c55366158fdf9c95df
Nov 03 03:49:38.739915 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.739880    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:49:38.740007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.739948    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:49:38.740007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.739983    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:49:38.833804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:38.833763    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:39.126840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:39.126799    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" event={"ID":"a4f5990c-30ac-495d-851a-a865a50e5286","Type":"ContainerStarted","Data":"87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e"}
Nov 03 03:49:39.127243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:39.126859    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" event={"ID":"a4f5990c-30ac-495d-851a-a865a50e5286","Type":"ContainerStarted","Data":"af8b41fafd8eb3a3869281cf6a4c5800ef9c0cedf14605c55366158fdf9c95df"}
Nov 03 03:49:39.249888 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:39.249828    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podStartSLOduration=2.249813255 podStartE2EDuration="2.249813255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:49:39.248223887 +0000 UTC m=+2870.479116318" watchObservedRunningTime="2026-11-03 03:49:39.249813255 +0000 UTC m=+2870.480705745"
Nov 03 03:49:39.304576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:39.298653    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:39.304576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:39.300025    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:39.304576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:39.300076    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:40.298098 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:40.298065    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:40.298532 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:40.298117    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:41.297508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:41.297477    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:41.297770 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:41.297749    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:42.304040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:42.303992    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:42.304515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:42.304061    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:43.297537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:43.297506    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:43.297763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:43.297675    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:43.736233 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:43.736200    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:44.297493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:44.297453    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:44.297681 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:44.297510    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:45.298162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:45.298114    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 03:49:45.298737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:45.298203    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 03:49:46.302330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:46.302300    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:46.302848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:46.302782    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:46.303511 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:46.303491    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 03:49:46.392541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:46.392508    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:48.632977 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:48.632939    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:50.984820 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:49:50.984739    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:49:53.551648 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:53.551536    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:53.697689 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:53.697659    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:55.305830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:55.305796    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:49:57.743080 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:57.743042    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:58.059657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:58.059583    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:58.607821 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:58.607793    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:59.328070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:59.328038    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:49:59.472841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:49:59.472813    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:00.782039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.782007    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"]
Nov 03 03:50:00.786246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.786223    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.832831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.832802    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"]
Nov 03 03:50:00.836788 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.836771    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.885680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.885648    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"]
Nov 03 03:50:00.936770 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.936400    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"]
Nov 03 03:50:00.951843 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.951812    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.951857    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.951926    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.951976    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952041    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952068    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952138 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952104    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952154    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952178    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952264    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952289    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952307    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952386    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952411    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952457    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952473    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:00.952515 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952491    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nhn\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:00.952663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:00.952595    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbjh\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.053995    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054029    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054069 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054049    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nhn\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054073    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbjh\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054111    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054150    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054192    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054220    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054260    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054295    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054331    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054396    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054463    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054570    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054620    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054683    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054750    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054943    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.054971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.054950    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.055336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055057    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.055336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055088    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.055336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055130    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.055336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055136    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.055336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055154    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.055654 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055368    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.055718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055673    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.055718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055706    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.056021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.055992    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.057190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.057145    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.057358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.057334    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.057591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.057540    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.057802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.057783    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.123577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.117499    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nhn\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.125703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.125672    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.126504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.126480    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbjh\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.126774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.126756    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token\") pod \"e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") " pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.148819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.148798    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:01.399760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.399723    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:01.579573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.579531    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"]
Nov 03 03:50:01.581284 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:50:01.581259    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cc67dd_bb35_4e7a_be16_1e39dbdcb9f3.slice/crio-92b23774a5cbec1146b4274cefda6fe743f27f751165263fa5fe2e1dff02a900 WatchSource:0}: Error finding container 92b23774a5cbec1146b4274cefda6fe743f27f751165263fa5fe2e1dff02a900: Status 404 returned error can't find the container with id 92b23774a5cbec1146b4274cefda6fe743f27f751165263fa5fe2e1dff02a900
Nov 03 03:50:01.583750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.583717    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:01.583816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.583785    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:01.583855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.583819    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:01.778398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.778370    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"]
Nov 03 03:50:01.778939 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:50:01.778919    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b39e1f8_90c1_4954_865b_483226e9a0b0.slice/crio-c64df5f3371af7af222f7327b2ba5b8a5d3c899698f5888c3f8393462b922d7b WatchSource:0}: Error finding container c64df5f3371af7af222f7327b2ba5b8a5d3c899698f5888c3f8393462b922d7b: Status 404 returned error can't find the container with id c64df5f3371af7af222f7327b2ba5b8a5d3c899698f5888c3f8393462b922d7b
Nov 03 03:50:01.782203 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.782168    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:01.782503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.782236    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:01.782503 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:01.782267    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:02.267461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.265260    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" event={"ID":"1b39e1f8-90c1-4954-865b-483226e9a0b0","Type":"ContainerStarted","Data":"49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291"}
Nov 03 03:50:02.267461 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.265327    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" event={"ID":"1b39e1f8-90c1-4954-865b-483226e9a0b0","Type":"ContainerStarted","Data":"c64df5f3371af7af222f7327b2ba5b8a5d3c899698f5888c3f8393462b922d7b"}
Nov 03 03:50:02.271818 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.271787    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" event={"ID":"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3","Type":"ContainerStarted","Data":"e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c"}
Nov 03 03:50:02.272070 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.272046    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" event={"ID":"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3","Type":"ContainerStarted","Data":"92b23774a5cbec1146b4274cefda6fe743f27f751165263fa5fe2e1dff02a900"}
Nov 03 03:50:02.403573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.402611    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:02.403573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.402737    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:02.403573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.402781    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:02.415576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.408884    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podStartSLOduration=2.408863149 podStartE2EDuration="2.408863149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:02.392883578 +0000 UTC m=+2893.623776011" watchObservedRunningTime="2026-11-03 03:50:02.408863149 +0000 UTC m=+2893.639755569"
Nov 03 03:50:02.517250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:02.517189    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podStartSLOduration=2.517173986 podStartE2EDuration="2.517173986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:02.515991404 +0000 UTC m=+2893.746883822" watchObservedRunningTime="2026-11-03 03:50:02.517173986 +0000 UTC m=+2893.748066479"
Nov 03 03:50:03.148968 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:03.148935    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:03.150763 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:03.150728    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 03:50:03.150893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:03.150783    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 03:50:03.404639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:03.400972    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:03.404639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:03.401039    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:03.758187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:03.758111    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:04.149656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:04.149627    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 03:50:04.150072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:04.149676    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 03:50:04.277575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:04.276262    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:04.410668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:04.406530    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:04.410668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:04.406612    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:04.484522 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:04.484455    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:05.149878 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:05.149841    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 03:50:05.150280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:05.149906    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 03:50:05.410105 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:05.410004    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:05.410383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:05.410334    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:06.149635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:06.149604    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 03:50:06.149884 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:06.149856    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 03:50:06.408111 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:06.408002    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:06.408822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:06.408788    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:07.149883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:07.149844    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 03:50:07.150077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:07.149894    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 03:50:07.402969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:07.402873    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:07.403905 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:07.403874    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:08.149285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:08.149246    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 03 03:50:08.149538 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:08.149299    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 03 03:50:08.400950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:08.400855    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 03 03:50:08.400950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:08.400923    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 03 03:50:08.723355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:08.723256    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:09.154718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:09.154691    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:09.155172 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:09.155148    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:09.156024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:09.155998    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:50:09.404835 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:09.404755    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:09.405304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:09.405138    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:09.405695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:09.405677    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:50:10.711136 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:10.711095    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:13.626902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:13.626869    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:17.967906 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:17.967866    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"]
Nov 03 03:50:18.069988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.069952    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:18.075639 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:18.075615    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-serving-cert: secret "maas-api-e2e-praxis-b-e84ee2-serving-cert" not found
Nov 03 03:50:18.075872 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:18.075853    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls podName:e18ba05a-3609-4759-847b-6cc1a4a22d77 nodeName:}" failed. No retries permitted until 2026-11-03 03:50:18.575835435 +0000 UTC m=+2909.806727832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls") pod "maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77") : secret "maas-api-e2e-praxis-b-e84ee2-serving-cert" not found
Nov 03 03:50:18.437035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.436985    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" podUID="e18ba05a-3609-4759-847b-6cc1a4a22d77" containerName="maas-api" containerID="cri-o://7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599" gracePeriod=30
Nov 03 03:50:18.522502 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.522470    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:18.594229 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:18.593773    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-serving-cert: secret "maas-api-e2e-praxis-b-e84ee2-serving-cert" not found
Nov 03 03:50:18.594229 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:18.593852    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls podName:e18ba05a-3609-4759-847b-6cc1a4a22d77 nodeName:}" failed. No retries permitted until 2026-11-03 03:50:19.593832731 +0000 UTC m=+2910.824725149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls") pod "maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77") : secret "maas-api-e2e-praxis-b-e84ee2-serving-cert" not found
Nov 03 03:50:18.711757 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.711728    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:50:18.795726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.795700    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn\" (UniqueName: \"kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn\") pod \"e18ba05a-3609-4759-847b-6cc1a4a22d77\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") "
Nov 03 03:50:18.795855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.795756    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs\") pod \"e18ba05a-3609-4759-847b-6cc1a4a22d77\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") "
Nov 03 03:50:18.795904 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.795874    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\") pod \"e18ba05a-3609-4759-847b-6cc1a4a22d77\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") "
Nov 03 03:50:18.795944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.795917    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca\") pod \"e18ba05a-3609-4759-847b-6cc1a4a22d77\" (UID: \"e18ba05a-3609-4759-847b-6cc1a4a22d77\") "
Nov 03 03:50:18.796359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.796323    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca" pod "e18ba05a-3609-4759-847b-6cc1a4a22d77" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:50:18.796495 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.796476    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18ba05a-3609-4759-847b-6cc1a4a22d77-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:18.798084 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.798061    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs" pod "e18ba05a-3609-4759-847b-6cc1a4a22d77" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:50:18.798204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.798114    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls" pod "e18ba05a-3609-4759-847b-6cc1a4a22d77" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:50:18.798204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.798118    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn" pod "e18ba05a-3609-4759-847b-6cc1a4a22d77" (UID: "e18ba05a-3609-4759-847b-6cc1a4a22d77"). InnerVolumeSpecName "kube-api-access-66pkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:50:18.897327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.897296    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pkn\" (UniqueName: \"kubernetes.io/projected/e18ba05a-3609-4759-847b-6cc1a4a22d77-kube-api-access-66pkn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:18.897327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.897325    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:18.897523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:18.897338    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18ba05a-3609-4759-847b-6cc1a4a22d77-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:19.444594 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.444561    5515 generic.go:356] "Generic (PLEG): container finished" podID="e18ba05a-3609-4759-847b-6cc1a4a22d77" containerID="7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599" exitCode=0
Nov 03 03:50:19.445056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.444633    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"
Nov 03 03:50:19.445056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.444655    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" event={"ID":"e18ba05a-3609-4759-847b-6cc1a4a22d77","Type":"ContainerDied","Data":"7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599"}
Nov 03 03:50:19.445056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.444735    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x" event={"ID":"e18ba05a-3609-4759-847b-6cc1a4a22d77","Type":"ContainerDied","Data":"d9e8ac163a422584981d6ec6f753906715a97ffcf2cd86e76715cc7426968a1c"}
Nov 03 03:50:19.445056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.444766    5515 scope.go:122] "RemoveContainer" containerID="7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599"
Nov 03 03:50:19.454649 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.454631    5515 scope.go:122] "RemoveContainer" containerID="7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599"
Nov 03 03:50:19.454950 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:19.454919    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599\": container with ID starting with 7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599 not found: ID does not exist" containerID="7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599"
Nov 03 03:50:19.455066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.454959    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599"} err="failed to get container status \"7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599\": rpc error: code = NotFound desc = could not find container \"7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599\": container with ID starting with 7ea323c56904db146e5739241d594ab2e405b06ae8869a45ad3ae0f54c96e599 not found: ID does not exist"
Nov 03 03:50:19.613537 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.613502    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"]
Nov 03 03:50:19.619048 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.619022    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e84ee2-54c5db8fd5-hpk5x"]
Nov 03 03:50:19.967484 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:19.967457    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:20.894422 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:20.894387    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18ba05a-3609-4759-847b-6cc1a4a22d77" path="/var/lib/kubelet/pods/e18ba05a-3609-4759-847b-6cc1a4a22d77/volumes"
Nov 03 03:50:20.984483 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:20.984444    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:50:21.172465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:21.172376    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:22.404874 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.404847    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:22.625697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.625667    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:22.972383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.972348    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"]
Nov 03 03:50:22.972886 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.972872    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e18ba05a-3609-4759-847b-6cc1a4a22d77" containerName="maas-api"
Nov 03 03:50:22.972938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.972889    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18ba05a-3609-4759-847b-6cc1a4a22d77" containerName="maas-api"
Nov 03 03:50:22.972991 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.972986    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18ba05a-3609-4759-847b-6cc1a4a22d77" containerName="maas-api"
Nov 03 03:50:22.977306 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:22.977289    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.084268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.084232    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"]
Nov 03 03:50:23.155591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.155527    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.155759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.155658    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.155759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.155703    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.155759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.155736    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.155908 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.155825    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.155908 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.155845    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.156056 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.156033    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.156153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.156138    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.156211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.156186    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78kq\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257459    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257534 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257508    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257572    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257593    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257658    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257715    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.257775 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257739    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78kq\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257798    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257850    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258041 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.257996    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.258231    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.258235    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.258294    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.258403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.258382    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.260173 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.260146    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.260496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.260478    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.319238 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.319213    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78kq\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.319368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.319252    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token\") pod \"e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") " pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:23.590651 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:23.590578    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:24.034685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.034656    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"]
Nov 03 03:50:24.035743 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:50:24.035715    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff81363_5ff0_433f_b8b2_132e93a2765a.slice/crio-beadb6f3d0b53f9386925840017e9bb0b85c50c386b77744be83caf029e650fe WatchSource:0}: Error finding container beadb6f3d0b53f9386925840017e9bb0b85c50c386b77744be83caf029e650fe: Status 404 returned error can't find the container with id beadb6f3d0b53f9386925840017e9bb0b85c50c386b77744be83caf029e650fe
Nov 03 03:50:24.038620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.038587    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:24.038712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.038652    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:24.038712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.038680    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:50:24.100196 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.100174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:24.488039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.487991    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" event={"ID":"eff81363-5ff0-433f-b8b2-132e93a2765a","Type":"ContainerStarted","Data":"2effd1b0fda5abe5b1880e5842bdbc49a519684690f4ed5e10cfdfa69d3ae1bf"}
Nov 03 03:50:24.488243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.488071    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" event={"ID":"eff81363-5ff0-433f-b8b2-132e93a2765a","Type":"ContainerStarted","Data":"beadb6f3d0b53f9386925840017e9bb0b85c50c386b77744be83caf029e650fe"}
Nov 03 03:50:24.595574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.591050    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:24.614600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.612259    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podStartSLOduration=2.612243303 podStartE2EDuration="2.612243303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:50:24.611464469 +0000 UTC m=+2915.842356888" watchObservedRunningTime="2026-11-03 03:50:24.612243303 +0000 UTC m=+2915.843135723"
Nov 03 03:50:24.632169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:24.632139    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:25.035012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:25.034979    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:50:25.181007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:25.180971    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:25.593523 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:25.593481    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:25.593708 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:25.593536    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:26.594919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:26.594879    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:26.596350 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:26.596319    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:26.993645 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:26.936700    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:27.628601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:27.627697    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:27.628601 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:27.627764    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:28.161362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.160625    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:28.189949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.189582    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 03:50:28.322832 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.321640    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 03:50:28.377688 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.377640    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:28.481638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.475222    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:28.537655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.537616    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:28.589087 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.589045    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:28.606783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.606710    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:28.607145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.607099    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:28.633295 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.633259    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded"
Nov 03 03:50:28.660683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:28.660633    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:29.193744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:29.193689    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:29.194969 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:29.194946    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:50:29.290581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:29.288207    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 03:50:29.592407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:29.592319    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:29.593151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:29.593109    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:30.591150 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:30.591115    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:30.591597 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:30.591172    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:30.632024 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:30.631978    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:31.591395 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:31.591355    5515 patch_prober.go:29] interesting pod/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 03 03:50:31.591924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:31.591432    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 03 03:50:33.532953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:33.532918    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:33.533721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:33.533684    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:33.534639 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:33.534614    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 03:50:33.680813 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:33.680776    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:35.629636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:35.629602    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:40.565311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:40.565280    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:44.848148 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:44.848111    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:46.012057 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:46.012026    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:47.468825 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:47.468787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:48.081630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:48.081592    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"]
Nov 03 03:50:48.082112 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:48.082082    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" containerID="cri-o://861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b" gracePeriod=30
Nov 03 03:50:48.082326 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:48.082080    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:48.227450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:48.227414    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:48.907283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:48.907207    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:49.079467 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:49.079388    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:49.932576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:49.922633    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:50.114750 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:50.112631    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 03:50:50.313632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:50.308586    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:50.425819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:50.425781    5515 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded"
Nov 03 03:50:50.456001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:50.455962    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:50.711407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:50.711371    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:51.051447 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:51.051339    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:50:51.221996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:51.221965    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:52.577383 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:52.577343    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:52.950304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:52.950252    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:53.094344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.094304    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:53.273898 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.269106    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:53.525133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.524999    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:53.576276 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.576247    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:50:53.738491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.738457    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:53.746635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746609    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.746785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746668    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.746785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746696    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.746785 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746752    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.746956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746783    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.746956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746916    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:50:53.747094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.746987    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.747183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747126    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:50:53.747183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747140    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:50:53.747268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747255    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:50:53.747470 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747447    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.747593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747490    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.747824 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747805    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo\") pod \"4ad47c06-3727-4e24-93f4-7137af0321ee\" (UID: \"4ad47c06-3727-4e24-93f4-7137af0321ee\") "
Nov 03 03:50:53.747919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.747829    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:50:53.748517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.748495    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.748710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.748686    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.748710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.748711    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.748902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.748725    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad47c06-3727-4e24-93f4-7137af0321ee-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.748902 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.748739    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.749664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.749641    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "kube-api-access-xjn8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:50:53.750204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.750184    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:50:53.750364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.750337    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:50:53.750475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.750375    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token" pod "4ad47c06-3727-4e24-93f4-7137af0321ee" (UID: "4ad47c06-3727-4e24-93f4-7137af0321ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:50:53.849993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.849910    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjn8r\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-kube-api-access-xjn8r\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.849993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.849940    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.849993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.849949    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.849993 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.849957    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad47c06-3727-4e24-93f4-7137af0321ee-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:50:53.881405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.881370    5515 generic.go:356] "Generic (PLEG): container finished" podID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerID="861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b" exitCode=0
Nov 03 03:50:53.881586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.881422    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"
Nov 03 03:50:53.881586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.881445    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" event={"ID":"4ad47c06-3727-4e24-93f4-7137af0321ee","Type":"ContainerDied","Data":"861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b"}
Nov 03 03:50:53.881586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.881502    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" event={"ID":"4ad47c06-3727-4e24-93f4-7137af0321ee","Type":"ContainerDied","Data":"c17aebd1ec36baf52092184bc6755d9eab22160fc7b5eab5ffa4f74662719c10"}
Nov 03 03:50:53.881586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.881524    5515 scope.go:122] "RemoveContainer" containerID="861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b"
Nov 03 03:50:53.892035 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.892016    5515 scope.go:122] "RemoveContainer" containerID="3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"
Nov 03 03:50:53.901767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.901744    5515 scope.go:122] "RemoveContainer" containerID="861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b"
Nov 03 03:50:53.902035 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:53.902016    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b\": container with ID starting with 861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b not found: ID does not exist" containerID="861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b"
Nov 03 03:50:53.902121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.902048    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b"} err="failed to get container status \"861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b\": rpc error: code = NotFound desc = could not find container \"861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b\": container with ID starting with 861b0d92461bf86173203de56de8db845c621d43de1fad2ec4576007828c219b not found: ID does not exist"
Nov 03 03:50:53.902121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.902071    5515 scope.go:122] "RemoveContainer" containerID="3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"
Nov 03 03:50:53.902302 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:50:53.902283    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd\": container with ID starting with 3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd not found: ID does not exist" containerID="3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"
Nov 03 03:50:53.902355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:53.902310    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd"} err="failed to get container status \"3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd\": rpc error: code = NotFound desc = could not find container \"3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd\": container with ID starting with 3f41fc1f08a371eb78b8817f85543bffb673f4bd115eec9269b19de9eb3814dd not found: ID does not exist"
Nov 03 03:50:54.052288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.052257    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"]
Nov 03 03:50:54.057703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.057677    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk"]
Nov 03 03:50:54.426206 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.426173    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:54.534217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.534177    5515 patch_prober.go:29] interesting pod/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:50:54.534407 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.534277    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-29dd3f-openshift-default-5767c8d5-mxfkk" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:50:54.574961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.574930    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:54.893473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:54.893440    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" path="/var/lib/kubelet/pods/4ad47c06-3727-4e24-93f4-7137af0321ee/volumes"
Nov 03 03:50:55.710712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:55.710678    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:57.144139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:57.144107    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:58.650365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:58.650332    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:58.799273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:58.799242    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:58.956301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:58.956220    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:59.106341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:59.106309    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:59.480231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:59.480037    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:50:59.717972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:50:59.717917    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:01.922868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:01.922836    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:02.193666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:02.193539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:02.340794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:02.340759    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:03.407027 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.406995    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:03.477487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.477457    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"]
Nov 03 03:51:03.477703 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.477683    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" podUID="f590246a-4c24-48da-bed0-36bc2ef38d8b" containerName="maas-api" containerID="cri-o://7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42" gracePeriod=30
Nov 03 03:51:03.581829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.581796    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:03.734267 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.734242    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:51:03.742299 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.742273    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:03.747611 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.747588    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca\") pod \"f590246a-4c24-48da-bed0-36bc2ef38d8b\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") "
Nov 03 03:51:03.747734 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.747707    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\") pod \"f590246a-4c24-48da-bed0-36bc2ef38d8b\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") "
Nov 03 03:51:03.747808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.747763    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs\") pod \"f590246a-4c24-48da-bed0-36bc2ef38d8b\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") "
Nov 03 03:51:03.747864 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.747829    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j\" (UniqueName: \"kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j\") pod \"f590246a-4c24-48da-bed0-36bc2ef38d8b\" (UID: \"f590246a-4c24-48da-bed0-36bc2ef38d8b\") "
Nov 03 03:51:03.748014 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.747984    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca" pod "f590246a-4c24-48da-bed0-36bc2ef38d8b" (UID: "f590246a-4c24-48da-bed0-36bc2ef38d8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:51:03.748414 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.748363    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f590246a-4c24-48da-bed0-36bc2ef38d8b-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:03.750143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.750121    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs" pod "f590246a-4c24-48da-bed0-36bc2ef38d8b" (UID: "f590246a-4c24-48da-bed0-36bc2ef38d8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:51:03.750524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.750497    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls" pod "f590246a-4c24-48da-bed0-36bc2ef38d8b" (UID: "f590246a-4c24-48da-bed0-36bc2ef38d8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:51:03.750524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.750502    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j" pod "f590246a-4c24-48da-bed0-36bc2ef38d8b" (UID: "f590246a-4c24-48da-bed0-36bc2ef38d8b"). InnerVolumeSpecName "kube-api-access-xb58j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:51:03.851271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.851239    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:03.851271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.851265    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f590246a-4c24-48da-bed0-36bc2ef38d8b-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:03.851271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.851276    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb58j\" (UniqueName: \"kubernetes.io/projected/f590246a-4c24-48da-bed0-36bc2ef38d8b-kube-api-access-xb58j\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:03.965145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.965059    5515 generic.go:356] "Generic (PLEG): container finished" podID="f590246a-4c24-48da-bed0-36bc2ef38d8b" containerID="7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42" exitCode=0
Nov 03 03:51:03.965145 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.965118    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"
Nov 03 03:51:03.965330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.965146    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" event={"ID":"f590246a-4c24-48da-bed0-36bc2ef38d8b","Type":"ContainerDied","Data":"7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42"}
Nov 03 03:51:03.965330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.965197    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt" event={"ID":"f590246a-4c24-48da-bed0-36bc2ef38d8b","Type":"ContainerDied","Data":"ef84691d67bceaa45e4b69f269143f60aa67176f65289c308940e4556bb65f7b"}
Nov 03 03:51:03.965330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.965220    5515 scope.go:122] "RemoveContainer" containerID="7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42"
Nov 03 03:51:04.002747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.979483    5515 scope.go:122] "RemoveContainer" containerID="7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42"
Nov 03 03:51:04.002747 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:51:03.979900    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42\": container with ID starting with 7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42 not found: ID does not exist" containerID="7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42"
Nov 03 03:51:04.002747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:03.979937    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42"} err="failed to get container status \"7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42\": rpc error: code = NotFound desc = could not find container \"7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42\": container with ID starting with 7841124c3f10565df627c492f96a75c52b8fdb7115dfd85dca92127f0616bf42 not found: ID does not exist"
Nov 03 03:51:04.170526 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:04.170492    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"]
Nov 03 03:51:04.175089 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:04.175059    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9b4cb2-85fbfcbf66-vcpqt"]
Nov 03 03:51:04.722957 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:04.722920    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:04.892695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:04.892665    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:04.895633 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:04.895609    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f590246a-4c24-48da-bed0-36bc2ef38d8b" path="/var/lib/kubelet/pods/f590246a-4c24-48da-bed0-36bc2ef38d8b/volumes"
Nov 03 03:51:05.129675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.129642    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:05.539958 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.539881    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:05.599081 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599050    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"]
Nov 03 03:51:05.599604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599587    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f590246a-4c24-48da-bed0-36bc2ef38d8b" containerName="maas-api"
Nov 03 03:51:05.599604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599604    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f590246a-4c24-48da-bed0-36bc2ef38d8b" containerName="maas-api"
Nov 03 03:51:05.599709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599628    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy"
Nov 03 03:51:05.599709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599633    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy"
Nov 03 03:51:05.599709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599656    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy"
Nov 03 03:51:05.599709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599661    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy"
Nov 03 03:51:05.599842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599745    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy"
Nov 03 03:51:05.599842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.599760    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f590246a-4c24-48da-bed0-36bc2ef38d8b" containerName="maas-api"
Nov 03 03:51:05.604405 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.604385    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.659114 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.659078    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-66d902-openshift-default-dockercfg-5zpwp\""
Nov 03 03:51:05.688352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.688306    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.690205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.690175    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.690490 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.690472    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqtkl\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.690965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.690940    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.710598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.691180    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.741133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.741076    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.744849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.744824    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.745431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.745407    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.745737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.745716    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.746142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.741662    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:05.756222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.756124    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"]
Nov 03 03:51:05.846914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.846832    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.846914 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.846888    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.846931    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.846988    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847057    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847129    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847186    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847227    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqtkl\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847277    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847433    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.847713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.847695    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.848077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.848019    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.848077 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.848048    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.848249 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.848218    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.850475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.850457    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.851142 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.851125    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.909022 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.908997    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqtkl\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.909194 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.908993    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token\") pod \"e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") " pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:05.918988 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:05.918958    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:06.042620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.042467    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:06.350403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.350372    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"]
Nov 03 03:51:06.351487 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:06.351458    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517b403f_ed14_4aa2_8aee_3943ac901a12.slice/crio-caa9abf2c973b782dd23e516b9634ccaf4e6ff185b735cd59a8dd1c75d8cda74 WatchSource:0}: Error finding container caa9abf2c973b782dd23e516b9634ccaf4e6ff185b735cd59a8dd1c75d8cda74: Status 404 returned error can't find the container with id caa9abf2c973b782dd23e516b9634ccaf4e6ff185b735cd59a8dd1c75d8cda74
Nov 03 03:51:06.353997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.353956    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:06.354094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.354030    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:06.354094 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.354066    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:06.990831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.990788    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" event={"ID":"517b403f-ed14-4aa2-8aee-3943ac901a12","Type":"ContainerStarted","Data":"72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"}
Nov 03 03:51:06.991264 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:06.990852    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" event={"ID":"517b403f-ed14-4aa2-8aee-3943ac901a12","Type":"ContainerStarted","Data":"caa9abf2c973b782dd23e516b9634ccaf4e6ff185b735cd59a8dd1c75d8cda74"}
Nov 03 03:51:07.113709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:07.113660    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podStartSLOduration=2.11364616 podStartE2EDuration="2.11364616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:07.112913453 +0000 UTC m=+2958.343805870" watchObservedRunningTime="2026-11-03 03:51:07.11364616 +0000 UTC m=+2958.344538578"
Nov 03 03:51:07.920013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:07.919970    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:07.925646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:07.925624    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:07.996222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:07.996189    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:07.997280 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:07.997260    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:51:08.644697 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:08.644646    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:08.901709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:08.901522    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:09.131795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.131703    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:09.286842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.286755    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:09.380412 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.380383    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"]
Nov 03 03:51:09.381038 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.381023    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad47c06-3727-4e24-93f4-7137af0321ee" containerName="istio-proxy"
Nov 03 03:51:09.384341 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.384325    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.434388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.434359    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.434520 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.434429    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2dx\" (UniqueName: \"kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx\") pod \"payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.439370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.439348    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-10120d\""
Nov 03 03:51:09.439493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.439391    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-10120d-dockercfg-pvzt2\""
Nov 03 03:51:09.443157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.443136    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"]
Nov 03 03:51:09.585638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.578580    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.585638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.578880    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2dx\" (UniqueName: \"kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx\") pod \"payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.585638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.579991    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.642030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.642001    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2dx\" (UniqueName: \"kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx\") pod \"payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:09.698476 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:09.698446    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:10.071980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.071951    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"]
Nov 03 03:51:10.073255 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:10.073232    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415ade92_351b_4c3f_b432_3d43af90403d.slice/crio-ca76232579a21952f1da753685b15d7f0a923d4a31188bcad824694c3aae94f6 WatchSource:0}: Error finding container ca76232579a21952f1da753685b15d7f0a923d4a31188bcad824694c3aae94f6: Status 404 returned error can't find the container with id ca76232579a21952f1da753685b15d7f0a923d4a31188bcad824694c3aae94f6
Nov 03 03:51:10.337016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.336939    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:10.539121 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.539092    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:10.684462 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.684428    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:10.820415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.820380    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"]
Nov 03 03:51:10.820683 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.820662    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy" containerID="cri-o://2b8095a16e051ff3ecd4d6a428f465dc4506b5c10cdc45f642bec7478bce25d3" gracePeriod=30
Nov 03 03:51:10.839156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.839101    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:10.986631 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:10.985819    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:11.014384 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:11.014345    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" event={"ID":"415ade92-351b-4c3f-b432-3d43af90403d","Type":"ContainerStarted","Data":"7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9"}
Nov 03 03:51:11.014609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:11.014410    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" event={"ID":"415ade92-351b-4c3f-b432-3d43af90403d","Type":"ContainerStarted","Data":"ca76232579a21952f1da753685b15d7f0a923d4a31188bcad824694c3aae94f6"}
Nov 03 03:51:11.014609 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:11.014584    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:11.134232 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:11.134175    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" podStartSLOduration=2.134161197 podStartE2EDuration="2.134161197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:11.131195135 +0000 UTC m=+2962.362087565" watchObservedRunningTime="2026-11-03 03:51:11.134161197 +0000 UTC m=+2962.365053669"
Nov 03 03:51:11.142879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:11.142830    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:12.888450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:12.888414    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:13.409374 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.409340    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"]
Nov 03 03:51:13.413109 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.413086    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.468378 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.468352    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-efa9bf-openshift-default-dockercfg-zx2xx\""
Nov 03 03:51:13.469275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469253    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469288    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469471 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469412    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469467    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469639    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469674    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469735    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469820 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469807    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.469895 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.469882    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nxh\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.507208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.507174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:13.521613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.521583    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"]
Nov 03 03:51:13.576445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.576359    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.583782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.583741    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.607239    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.607308    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.607449    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.607494    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nxh\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.607531    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.607157    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.584421    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.608574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.608096    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.609160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.608659    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.609160 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.608937    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.609288 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.609218    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.624026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.609674    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.624026 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.610393    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.637572 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.629896    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.697616 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.697453    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.703584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.699627    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:13.703584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.700457    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nxh\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh\") pod \"e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") " pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:13.730054 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:13.729127    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:14.172335 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:14.172310    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"]
Nov 03 03:51:14.173521 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:14.173488    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8a6bb3_970e_4fce_bcad_01233bf889fc.slice/crio-6edc13000317f84e55548d3d19e5f9921228704181453b9331ceaee1f9ed84c1 WatchSource:0}: Error finding container 6edc13000317f84e55548d3d19e5f9921228704181453b9331ceaee1f9ed84c1: Status 404 returned error can't find the container with id 6edc13000317f84e55548d3d19e5f9921228704181453b9331ceaee1f9ed84c1
Nov 03 03:51:14.176134 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:14.176102    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:14.176217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:14.176166    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:14.176217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:14.176197    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:15.041707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:15.041668    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" event={"ID":"cb8a6bb3-970e-4fce-bcad-01233bf889fc","Type":"ContainerStarted","Data":"3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"}
Nov 03 03:51:15.041926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:15.041728    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" event={"ID":"cb8a6bb3-970e-4fce-bcad-01233bf889fc","Type":"ContainerStarted","Data":"6edc13000317f84e55548d3d19e5f9921228704181453b9331ceaee1f9ed84c1"}
Nov 03 03:51:15.233795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:15.233744    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podStartSLOduration=2.233730904 podStartE2EDuration="2.233730904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:15.215429593 +0000 UTC m=+2966.446322014" watchObservedRunningTime="2026-11-03 03:51:15.233730904 +0000 UTC m=+2966.464623321"
Nov 03 03:51:15.729623 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:15.729594    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:15.734771 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:15.734748    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:16.051009 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.050978    5515 generic.go:356] "Generic (PLEG): container finished" podID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerID="2b8095a16e051ff3ecd4d6a428f465dc4506b5c10cdc45f642bec7478bce25d3" exitCode=0
Nov 03 03:51:16.051180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.051053    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" event={"ID":"509c0d87-f3b2-40e8-a21c-0ca99d6ae651","Type":"ContainerDied","Data":"2b8095a16e051ff3ecd4d6a428f465dc4506b5c10cdc45f642bec7478bce25d3"}
Nov 03 03:51:16.051180 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.051113    5515 scope.go:122] "RemoveContainer" containerID="42d8efa9f0ab2c9247a98fe6784ea9e7c4fcb00dc93700e08bab4a14ed48b43c"
Nov 03 03:51:16.051392 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.051369    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:16.052458 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.052435    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:51:16.092802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.092774    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:51:16.160103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160066    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160103 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160109    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160143    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160166    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160275    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160295    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160327    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160371    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:51:16.160678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160398    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160451    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo\") pod \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\" (UID: \"509c0d87-f3b2-40e8-a21c-0ca99d6ae651\") "
Nov 03 03:51:16.160678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160599    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:51:16.160678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160670    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:51:16.160942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160775    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:51:16.161050 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.160972    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:51:16.161285 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.161265    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.161361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.161287    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.161361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.161300    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.161361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.161314    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.161361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.161337    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.162995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.162960    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:51:16.163246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.163223    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:51:16.163656 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.163633    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:51:16.164063 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.164044    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq" pod "509c0d87-f3b2-40e8-a21c-0ca99d6ae651" (UID: "509c0d87-f3b2-40e8-a21c-0ca99d6ae651"). InnerVolumeSpecName "kube-api-access-xpgvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:51:16.263187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.263099    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.263187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.263138    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.263187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.263154    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpgvq\" (UniqueName: \"kubernetes.io/projected/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-kube-api-access-xpgvq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.263187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.263169    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/509c0d87-f3b2-40e8-a21c-0ca99d6ae651-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:16.316628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.316513    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:16.461984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.461952    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:16.639588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.634701    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:16.743113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.743072    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"]
Nov 03 03:51:16.743996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.743976    5515 cpu_manager.go:414] "containerMap: removing container" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy"
Nov 03 03:51:16.744096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.743997    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy"
Nov 03 03:51:16.744096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.744031    5515 cpu_manager.go:414] "containerMap: removing container" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy"
Nov 03 03:51:16.744096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.744039    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy"
Nov 03 03:51:16.744252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.744207    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy"
Nov 03 03:51:16.744252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.744223    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" containerName="istio-proxy"
Nov 03 03:51:16.748496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.748471    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:16.801857 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.801816    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-66d902\""
Nov 03 03:51:16.803795 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.803197    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-66d902-dockercfg-pk2k6\""
Nov 03 03:51:16.803967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.803939    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"]
Nov 03 03:51:16.896955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.896859    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2b7n\" (UniqueName: \"kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n\") pod \"payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:16.906762 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:16.906710    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:17.007489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.007454    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:17.007816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.007790    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2b7n\" (UniqueName: \"kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n\") pod \"payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:17.008307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.008290    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:17.058535 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.058507    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"
Nov 03 03:51:17.058699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.058578    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb" event={"ID":"509c0d87-f3b2-40e8-a21c-0ca99d6ae651","Type":"ContainerDied","Data":"e39c1bc0e4b2a38699a1d67dd672e0cd21dcf8a769770295e5bbb80ffbb8b842"}
Nov 03 03:51:17.058699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.058637    5515 scope.go:122] "RemoveContainer" containerID="2b8095a16e051ff3ecd4d6a428f465dc4506b5c10cdc45f642bec7478bce25d3"
Nov 03 03:51:17.069408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.069390    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2b7n\" (UniqueName: \"kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n\") pod \"payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:17.227718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.227633    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"]
Nov 03 03:51:17.230066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.230042    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9b4cb2-openshift-default-7784dd77c7-jr7lb"]
Nov 03 03:51:17.362696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.362666    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:17.738774 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.738744    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"]
Nov 03 03:51:17.739836 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:17.739813    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa654b1a_05f5_4154_b259_bc74b75cc1c9.slice/crio-bf12cdf1b549e28726b6c8b085fc21fd2102cdde717f8599b7a42799ac688aca WatchSource:0}: Error finding container bf12cdf1b549e28726b6c8b085fc21fd2102cdde717f8599b7a42799ac688aca: Status 404 returned error can't find the container with id bf12cdf1b549e28726b6c8b085fc21fd2102cdde717f8599b7a42799ac688aca
Nov 03 03:51:17.964347 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:17.964315    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:18.087510 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.087413    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" event={"ID":"aa654b1a-05f5-4154-b259-bc74b75cc1c9","Type":"ContainerStarted","Data":"2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee"}
Nov 03 03:51:18.087735 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.087510    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" event={"ID":"aa654b1a-05f5-4154-b259-bc74b75cc1c9","Type":"ContainerStarted","Data":"bf12cdf1b549e28726b6c8b085fc21fd2102cdde717f8599b7a42799ac688aca"}
Nov 03 03:51:18.087860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.087839    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:18.145533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.145468    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:18.206257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.206214    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" podStartSLOduration=2.206201144 podStartE2EDuration="2.206201144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:18.20491958 +0000 UTC m=+2969.435811998" watchObservedRunningTime="2026-11-03 03:51:18.206201144 +0000 UTC m=+2969.437093562"
Nov 03 03:51:18.303719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.303691    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:18.894812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:18.894783    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="509c0d87-f3b2-40e8-a21c-0ca99d6ae651" path="/var/lib/kubelet/pods/509c0d87-f3b2-40e8-a21c-0ca99d6ae651/volumes"
Nov 03 03:51:19.749910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:19.749872    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:51:21.171575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:21.171146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:21.226205 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:51:21.226127    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:51:21.353575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:21.337190    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:21.752574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:21.752348    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:22.018624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.014811    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"]
Nov 03 03:51:22.038575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.021000    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.053500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.046926    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q9m\" (UniqueName: \"kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m\") pod \"payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.053500 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.046997    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.148415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.148376    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"]
Nov 03 03:51:22.176587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.157955    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-efa9bf\""
Nov 03 03:51:22.308574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.294634    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:51:22.363573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.356775    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 03:51:22.404632 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.390896    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-efa9bf-dockercfg-2rxxw\""
Nov 03 03:51:22.404823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.404775    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q9m\" (UniqueName: \"kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m\") pod \"payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.404897 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.404858    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.445576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.441405    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.566624 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.565066    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q9m\" (UniqueName: \"kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m\") pod \"payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.664577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.661935    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw" podUID="e1aa2223-e7d4-4647-8be5-402eb71fed01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:51:22.720575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.719960    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 03:51:22.720762 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.720582    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:22.740254 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:22.729629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:23.321283 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:23.321253    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"]
Nov 03 03:51:23.322481 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:23.322453    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0a5e36_a7c0_44d5_b987_a79a18bac90b.slice/crio-5fe72f75202663ed1b081f7f99e81826373f9aab48eddb8b284616b5a9c41115 WatchSource:0}: Error finding container 5fe72f75202663ed1b081f7f99e81826373f9aab48eddb8b284616b5a9c41115: Status 404 returned error can't find the container with id 5fe72f75202663ed1b081f7f99e81826373f9aab48eddb8b284616b5a9c41115
Nov 03 03:51:23.481531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:23.481495    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" event={"ID":"cc0a5e36-a7c0-44d5-b987-a79a18bac90b","Type":"ContainerStarted","Data":"029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4"}
Nov 03 03:51:23.481719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:23.481591    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" event={"ID":"cc0a5e36-a7c0-44d5-b987-a79a18bac90b","Type":"ContainerStarted","Data":"5fe72f75202663ed1b081f7f99e81826373f9aab48eddb8b284616b5a9c41115"}
Nov 03 03:51:23.481719 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:23.481625    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:23.601192 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:23.601087    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" podStartSLOduration=2.601073364 podStartE2EDuration="2.601073364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:23.598928995 +0000 UTC m=+2974.829821435" watchObservedRunningTime="2026-11-03 03:51:23.601073364 +0000 UTC m=+2974.831965781"
Nov 03 03:51:27.743646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:27.743613    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:29.097465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:29.097435    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 03:51:31.872736 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:31.872704    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"]
Nov 03 03:51:31.873166 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:31.872887    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" podUID="a83c7e53-341c-42d8-864e-6c373311755e" containerName="maas-api" containerID="cri-o://e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd" gracePeriod=30
Nov 03 03:51:31.989068 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:31.988907    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" podUID="a83c7e53-341c-42d8-864e-6c373311755e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.175:8443/health\": dial tcp 10.134.0.175:8443: connect: connection refused"
Nov 03 03:51:32.010521 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.010493    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:32.196082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.196058    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:51:32.241151 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.241122    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs\") pod \"a83c7e53-341c-42d8-864e-6c373311755e\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") "
Nov 03 03:51:32.241293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.241276    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx\" (UniqueName: \"kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx\") pod \"a83c7e53-341c-42d8-864e-6c373311755e\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") "
Nov 03 03:51:32.241364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.241350    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\") pod \"a83c7e53-341c-42d8-864e-6c373311755e\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") "
Nov 03 03:51:32.241401 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.241381    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca\") pod \"a83c7e53-341c-42d8-864e-6c373311755e\" (UID: \"a83c7e53-341c-42d8-864e-6c373311755e\") "
Nov 03 03:51:32.241849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.241798    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca" pod "a83c7e53-341c-42d8-864e-6c373311755e" (UID: "a83c7e53-341c-42d8-864e-6c373311755e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:51:32.242298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.242275    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c7e53-341c-42d8-864e-6c373311755e-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:32.243486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.243459    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx" pod "a83c7e53-341c-42d8-864e-6c373311755e" (UID: "a83c7e53-341c-42d8-864e-6c373311755e"). InnerVolumeSpecName "kube-api-access-b94bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:51:32.243621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.243461    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs" pod "a83c7e53-341c-42d8-864e-6c373311755e" (UID: "a83c7e53-341c-42d8-864e-6c373311755e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:51:32.243621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.243519    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls" pod "a83c7e53-341c-42d8-864e-6c373311755e" (UID: "a83c7e53-341c-42d8-864e-6c373311755e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:51:32.344171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.344132    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:32.344171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.344172    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94bx\" (UniqueName: \"kubernetes.io/projected/a83c7e53-341c-42d8-864e-6c373311755e-kube-api-access-b94bx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:32.344438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.344189    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83c7e53-341c-42d8-864e-6c373311755e-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:51:32.547976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.547877    5515 generic.go:356] "Generic (PLEG): container finished" podID="a83c7e53-341c-42d8-864e-6c373311755e" containerID="e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd" exitCode=0
Nov 03 03:51:32.547976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.547911    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" event={"ID":"a83c7e53-341c-42d8-864e-6c373311755e","Type":"ContainerDied","Data":"e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd"}
Nov 03 03:51:32.547976 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.547944    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"
Nov 03 03:51:32.548268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.548002    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b" event={"ID":"a83c7e53-341c-42d8-864e-6c373311755e","Type":"ContainerDied","Data":"d9a63e70f754577a95dc92c86d9881d65286ac3b90058a12ff255070d543ae35"}
Nov 03 03:51:32.548268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.548039    5515 scope.go:122] "RemoveContainer" containerID="e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd"
Nov 03 03:51:32.559229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.559213    5515 scope.go:122] "RemoveContainer" containerID="e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd"
Nov 03 03:51:32.559488 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:51:32.559467    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd\": container with ID starting with e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd not found: ID does not exist" containerID="e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd"
Nov 03 03:51:32.559626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.559499    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd"} err="failed to get container status \"e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd\": rpc error: code = NotFound desc = could not find container \"e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd\": container with ID starting with e783195f5564db82c8eda063b3933d9219a402123eb64ad37cf81449d2edcccd not found: ID does not exist"
Nov 03 03:51:32.716387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.716352    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"]
Nov 03 03:51:32.722330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.722303    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8950af-8656794869-f8v6b"]
Nov 03 03:51:32.914457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:32.914423    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83c7e53-341c-42d8-864e-6c373311755e" path="/var/lib/kubelet/pods/a83c7e53-341c-42d8-864e-6c373311755e/volumes"
Nov 03 03:51:33.483051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.483024    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"]
Nov 03 03:51:33.483592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.483577    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a83c7e53-341c-42d8-864e-6c373311755e" containerName="maas-api"
Nov 03 03:51:33.483644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.483593    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c7e53-341c-42d8-864e-6c373311755e" containerName="maas-api"
Nov 03 03:51:33.483687 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.483682    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83c7e53-341c-42d8-864e-6c373311755e" containerName="maas-api"
Nov 03 03:51:33.488456 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.488438    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.543045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.543018    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-f3e02b-serving-cert\""
Nov 03 03:51:33.565911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.565886    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.566061 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.565968    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszvs\" (UniqueName: \"kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.566144 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.566118    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.566303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.566276    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.596518 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.596494    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"]
Nov 03 03:51:33.667638 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.667599    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.667831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.667688    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszvs\" (UniqueName: \"kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.667831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.667772    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.667955 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.667895    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.668079 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:51:33.668058    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-serving-cert: secret "maas-api-e2e-shared-a-w4-f3e02b-serving-cert" not found
Nov 03 03:51:33.668163 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:51:33.668150    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls podName:f2361be4-77ae-456b-99f6-80f3c5aa0aed nodeName:}" failed. No retries permitted until 2026-11-03 03:51:34.168130366 +0000 UTC m=+2985.399022779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls") pod "maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" (UID: "f2361be4-77ae-456b-99f6-80f3c5aa0aed") : secret "maas-api-e2e-shared-a-w4-f3e02b-serving-cert" not found
Nov 03 03:51:33.668367 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.668349    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.670435 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.670411    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.681352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.681328    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:33.758159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.757244    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszvs\" (UniqueName: \"kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:33.842584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.842524    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:33.989491 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:33.989445    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:34.181842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:34.181810    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:34.184362 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:34.184336    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:34.402658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:34.402629    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:34.489840 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:34.489816    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 03:51:34.783361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:34.783333    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"]
Nov 03 03:51:34.784102 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:34.784072    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2361be4_77ae_456b_99f6_80f3c5aa0aed.slice/crio-2cb332c7e911e7f8857218732bbb85705cdfdd41f440c57eef091b74f8f185e2 WatchSource:0}: Error finding container 2cb332c7e911e7f8857218732bbb85705cdfdd41f440c57eef091b74f8f185e2: Status 404 returned error can't find the container with id 2cb332c7e911e7f8857218732bbb85705cdfdd41f440c57eef091b74f8f185e2
Nov 03 03:51:34.785626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:34.785606    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:51:35.569322 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:35.569288    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" event={"ID":"f2361be4-77ae-456b-99f6-80f3c5aa0aed","Type":"ContainerStarted","Data":"2cb332c7e911e7f8857218732bbb85705cdfdd41f440c57eef091b74f8f185e2"}
Nov 03 03:51:35.684636 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:35.684596    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:35.847261 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:35.847156    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:36.609314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:36.609275    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" event={"ID":"f2361be4-77ae-456b-99f6-80f3c5aa0aed","Type":"ContainerStarted","Data":"d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc"}
Nov 03 03:51:36.615786 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:36.610349    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:36.730599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:36.730519    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" podStartSLOduration=2.5210155050000003 podStartE2EDuration="3.73050451s" totalImagesPullingTime="1.209489005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:36.72992558 +0000 UTC m=+2987.960817998" watchObservedRunningTime="2026-11-03 03:51:36.73050451 +0000 UTC m=+2987.961396928"
Nov 03 03:51:37.306162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:37.306134    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:37.813949 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:37.813881    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:38.940709 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:38.940677    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:39.871017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:39.870987    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:40.584279 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:40.584248    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:40.992417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:40.992380    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:42.621271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:42.621245    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 03:51:44.074819 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:44.074784    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:44.235671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:44.235639    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:44.381953 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:44.381919    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:45.669211 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:45.669179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:45.884751 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:45.884723    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:46.039108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:46.039019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:49.219421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.219386    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:49.403671 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.403629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:49.560508 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.551530    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:49.617961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.617927    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"]
Nov 03 03:51:49.622297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.622274    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.706849 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.706814    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn58s\" (UniqueName: \"kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s\") pod \"payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.707003 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.706945    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.740620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.725695    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-158d23\""
Nov 03 03:51:49.756387 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.750772    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"]
Nov 03 03:51:49.757799 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.752349    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-158d23-dockercfg-2dln7\""
Nov 03 03:51:49.808388 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.808292    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.810305 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.810103    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.810814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.810745    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn58s\" (UniqueName: \"kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s\") pod \"payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.823655 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.823617    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"]
Nov 03 03:51:49.829430 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.829028    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:49.847713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.847687    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"]
Nov 03 03:51:49.878592 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.878538    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn58s\" (UniqueName: \"kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s\") pod \"payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:49.913557 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.913523    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") " pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:49.913700 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.913659    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7blw\" (UniqueName: \"kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw\") pod \"payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") " pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:49.935214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:49.935188    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:50.015696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.015664    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") " pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:50.015841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.015773    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7blw\" (UniqueName: \"kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw\") pod \"payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") " pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:50.016737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.016718    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") " pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:50.077913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.077842    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7blw\" (UniqueName: \"kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw\") pod \"payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") " pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:50.129361 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.129338    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:51:50.131410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.131393    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:51:50.148419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.148396    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:50.161690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.161670    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:51:50.164336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.164318    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:51:50.312961 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.312937    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"]
Nov 03 03:51:50.314431 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:50.314407    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc837120_3a65_4b7a_99d9_82fe2665e12f.slice/crio-49bbcf2a5ed90dece4acd64e47af573ef8386833ef25af2be151b397cf5ddfeb WatchSource:0}: Error finding container 49bbcf2a5ed90dece4acd64e47af573ef8386833ef25af2be151b397cf5ddfeb: Status 404 returned error can't find the container with id 49bbcf2a5ed90dece4acd64e47af573ef8386833ef25af2be151b397cf5ddfeb
Nov 03 03:51:50.527127 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.527097    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"]
Nov 03 03:51:50.528045 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:50.528023    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacf57d0_664e_44ce_a75c_070023b37ac3.slice/crio-c65a579f7cd889e1a05c055e9bfbf691ab1714bbc5ad69a5062ab32f80958f13 WatchSource:0}: Error finding container c65a579f7cd889e1a05c055e9bfbf691ab1714bbc5ad69a5062ab32f80958f13: Status 404 returned error can't find the container with id c65a579f7cd889e1a05c055e9bfbf691ab1714bbc5ad69a5062ab32f80958f13
Nov 03 03:51:50.837956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.837873    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z" event={"ID":"cc837120-3a65-4b7a-99d9-82fe2665e12f","Type":"ContainerStarted","Data":"3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6"}
Nov 03 03:51:50.837956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.837930    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:51:50.837956 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.837951    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z" event={"ID":"cc837120-3a65-4b7a-99d9-82fe2665e12f","Type":"ContainerStarted","Data":"49bbcf2a5ed90dece4acd64e47af573ef8386833ef25af2be151b397cf5ddfeb"}
Nov 03 03:51:50.839563 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.839528    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf" event={"ID":"aacf57d0-664e-44ce-a75c-070023b37ac3","Type":"ContainerStarted","Data":"0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a"}
Nov 03 03:51:50.839675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.839609    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:51:50.839675 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.839628    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf" event={"ID":"aacf57d0-664e-44ce-a75c-070023b37ac3","Type":"ContainerStarted","Data":"c65a579f7cd889e1a05c055e9bfbf691ab1714bbc5ad69a5062ab32f80958f13"}
Nov 03 03:51:50.955247 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:50.955198    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z" podStartSLOduration=1.955184303 podStartE2EDuration="1.955184303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:50.954953237 +0000 UTC m=+3002.185845666" watchObservedRunningTime="2026-11-03 03:51:50.955184303 +0000 UTC m=+3002.186076720"
Nov 03 03:51:50.984727 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:51:50.984692    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:51:51.077591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:51.076862    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf" podStartSLOduration=2.076841674 podStartE2EDuration="2.076841674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:51.072103827 +0000 UTC m=+3002.302996247" watchObservedRunningTime="2026-11-03 03:51:51.076841674 +0000 UTC m=+3002.307734094"
Nov 03 03:51:52.960272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:52.960241    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"]
Nov 03 03:51:52.965612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:52.965593    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.063845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.063774    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064072    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064121    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064153    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064218    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064340 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064262    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpprg\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064519    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064659    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.064794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.064772    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.072930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.072905    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"]
Nov 03 03:51:53.165944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.165910    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166055    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166076    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166093    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166146 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166129    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166205    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpprg\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166265    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166297    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166341    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166365    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166386 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166382    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166466    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.166773 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.166748    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.167053 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.167030    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.168723 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.168699    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.169049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.169026    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.228299 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.228234    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.228458 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.228436    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpprg\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg\") pod \"e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") " pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.280021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.279985    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:53.715049 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.715023    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"]
Nov 03 03:51:53.716307 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:51:53.716267    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded637db4_0648_4435_ad95_0fa224eeed5a.slice/crio-43970f3fa838786634ce3ead5b419630cb09e0c063b8daf8976f4da1c23878cc WatchSource:0}: Error finding container 43970f3fa838786634ce3ead5b419630cb09e0c063b8daf8976f4da1c23878cc: Status 404 returned error can't find the container with id 43970f3fa838786634ce3ead5b419630cb09e0c063b8daf8976f4da1c23878cc
Nov 03 03:51:53.718823 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.718789    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:53.718911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.718862    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:53.718911 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.718891    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:51:53.860845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.860811    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" event={"ID":"ed637db4-0648-4435-ad95-0fa224eeed5a","Type":"ContainerStarted","Data":"ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79"}
Nov 03 03:51:53.861082 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.860861    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" event={"ID":"ed637db4-0648-4435-ad95-0fa224eeed5a","Type":"ContainerStarted","Data":"43970f3fa838786634ce3ead5b419630cb09e0c063b8daf8976f4da1c23878cc"}
Nov 03 03:51:53.983629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:53.983500    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podStartSLOduration=1.9834868650000002 podStartE2EDuration="1.983486865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:51:53.982796516 +0000 UTC m=+3005.213688936" watchObservedRunningTime="2026-11-03 03:51:53.983486865 +0000 UTC m=+3005.214379290"
Nov 03 03:51:54.174933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:54.174885    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:54.284595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:54.284496    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:51:54.285978 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:54.285948    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:51:54.286143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:54.286004    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:51:54.336621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:54.336572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:54.521893 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:54.521857    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:55.286577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:55.283851    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:51:55.286577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:55.283914    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:51:56.288485 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:56.288129    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:51:56.288485 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:56.288186    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:51:57.287019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:57.286980    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:51:57.287229 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:57.287036    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:51:58.285574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:58.282912    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:51:58.285574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:58.282988    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:51:59.232746 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:59.232710    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:51:59.283575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:59.281903    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:51:59.283575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:51:59.281964    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:52:00.280565 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:00.280511    5515 patch_prober.go:29] interesting pod/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 03:52:00.280960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:00.280594    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 03:52:01.284780 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:01.284750    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:52:01.285235 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:01.285214    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:52:01.286011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:01.285993    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:52:01.849593 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:01.849535    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:52:01.849766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:01.849641    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:52:02.779010 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:02.778976    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:03.873358 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.873328    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"]
Nov 03 03:52:03.877342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.877325    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:03.921756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.921720    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:03.921919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.921813    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:03.922171 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.922143    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjh5\" (UniqueName: \"kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:03.922311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.922297    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:03.932481 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.932364    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-efa9bf-serving-cert\""
Nov 03 03:52:03.986046 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:03.986015    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"]
Nov 03 03:52:04.023994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.023963    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.024179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.024108    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjh5\" (UniqueName: \"kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.024259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.024198    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.024259 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.024244    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.024363 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:52:04.024304    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-serving-cert: secret "maas-api-e2e-shared-a-w0-efa9bf-serving-cert" not found
Nov 03 03:52:04.024424 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:52:04.024384    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls podName:428d296d-b09c-4d16-8486-7f12791481bb nodeName:}" failed. No retries permitted until 2026-11-03 03:52:04.524361498 +0000 UTC m=+3015.755253900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls") pod "maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" (UID: "428d296d-b09c-4d16-8486-7f12791481bb") : secret "maas-api-e2e-shared-a-w0-efa9bf-serving-cert" not found
Nov 03 03:52:04.024808 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.024784    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.027075 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.027052    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.074243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.074219    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:04.086663 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.086635    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjh5\" (UniqueName: \"kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.230879 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.230729    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:04.375794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.375766    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:04.538113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.538018    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.540581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.540540    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:04.795798 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:04.795721    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:05.048497 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:05.048409    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:05.173686 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:05.173663    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"]
Nov 03 03:52:05.174680 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:52:05.174646    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428d296d_b09c_4d16_8486_7f12791481bb.slice/crio-dc5e50ab238d0a355b5fef9d8f0680c17bea08a5c7b997571b910a5d02458176 WatchSource:0}: Error finding container dc5e50ab238d0a355b5fef9d8f0680c17bea08a5c7b997571b910a5d02458176: Status 404 returned error can't find the container with id dc5e50ab238d0a355b5fef9d8f0680c17bea08a5c7b997571b910a5d02458176
Nov 03 03:52:05.956205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:05.956169    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" event={"ID":"428d296d-b09c-4d16-8486-7f12791481bb","Type":"ContainerStarted","Data":"dc5e50ab238d0a355b5fef9d8f0680c17bea08a5c7b997571b910a5d02458176"}
Nov 03 03:52:07.002814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:07.002774    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" event={"ID":"428d296d-b09c-4d16-8486-7f12791481bb","Type":"ContainerStarted","Data":"6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd"}
Nov 03 03:52:07.005524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:07.005502    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:07.127097 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:07.127042    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" podStartSLOduration=3.194493162 podStartE2EDuration="4.127027153s" totalImagesPullingTime="932.533991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:07.124442782 +0000 UTC m=+3018.355335200" watchObservedRunningTime="2026-11-03 03:52:07.127027153 +0000 UTC m=+3018.357919632"
Nov 03 03:52:09.123600 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:09.123566    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:11.875845 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:11.875803    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:52:14.017159 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:14.017121    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:14.019890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:14.019868    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 03:52:18.855792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:18.855752    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:20.986082 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:52:20.986039    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:52:22.997918 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:22.997882    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:22.998421 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:22.997968    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:23.122604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.122540    5515 generic.go:356] "Generic (PLEG): container finished" podID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerID="72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82" exitCode=0
Nov 03 03:52:23.122604 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.122594    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" event={"ID":"517b403f-ed14-4aa2-8aee-3943ac901a12","Type":"ContainerDied","Data":"72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"}
Nov 03 03:52:23.123004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.122990    5515 scope.go:122] "RemoveContainer" containerID="72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"
Nov 03 03:52:23.124602 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.124569    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:23.124689 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.124661    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:23.124727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.124709    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:23.664562 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:23.664521    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:24.129980 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:24.129947    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" event={"ID":"517b403f-ed14-4aa2-8aee-3943ac901a12","Type":"ContainerStarted","Data":"538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43"}
Nov 03 03:52:24.919776 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:24.919747    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:52:24.919966 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:24.919930    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:24.920021 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:24.919971    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:25.919804 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:25.919768    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:25.920205 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:25.919831    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:26.920028 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:26.919987    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:26.920438 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:26.920053    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:27.937578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:27.919956    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:27.937578 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:27.920047    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:28.547037 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:28.547005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:28.919349 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:28.919313    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:28.919543 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:28.919369    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:29.919680 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:29.919626    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Nov 03 03:52:29.920104 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:29.919714    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Nov 03 03:52:30.760712 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:30.760675    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:30.924466 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:30.924437    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:52:30.924943 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:30.924922    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:52:30.925767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:30.925747    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 03:52:31.250919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:31.250885    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:31.904658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:31.904613    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"]
Nov 03 03:52:31.917667 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:31.911980    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.019807 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.019775    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7ed2d6-openshift-default-dockercfg-lgh27\""
Nov 03 03:52:32.037637 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.028712    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"]
Nov 03 03:52:32.057282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.057215    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:32.110571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.086608    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.135721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.135677    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.136174 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.136145    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.136469 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.136447    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rg2\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.136759 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.136736    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.136909 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.136892    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.137007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.136931    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.137007 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.136991    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.137113 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.137037    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238205    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238266    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238297    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238339    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238377    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238435    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238480    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238536    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238759    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238839    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.238926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238871    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.239031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238934    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rg2\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.239031 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.238953    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.239266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.239246    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.240912 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.240887    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.241419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.241397    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.332067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.317393    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:32.332067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.330292    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.332067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.331998    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rg2\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2\") pod \"e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") " pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.438530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.438495    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:32.897227 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:52:32.897200    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9710500_d6bf_45c1_8724_8dcf08b046a5.slice/crio-edc5ce6fdd118e4a437c9195127d449f5f9511de882b2868af2196d45683c8c3 WatchSource:0}: Error finding container edc5ce6fdd118e4a437c9195127d449f5f9511de882b2868af2196d45683c8c3: Status 404 returned error can't find the container with id edc5ce6fdd118e4a437c9195127d449f5f9511de882b2868af2196d45683c8c3
Nov 03 03:52:32.899612 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.899580    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:32.899707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.899646    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:32.899707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.899675    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:32.937585 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:32.937543    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"]
Nov 03 03:52:33.234974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.234888    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" event={"ID":"b9710500-d6bf-45c1-8724-8dcf08b046a5","Type":"ContainerStarted","Data":"b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"}
Nov 03 03:52:33.234974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.234948    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" event={"ID":"b9710500-d6bf-45c1-8724-8dcf08b046a5","Type":"ContainerStarted","Data":"edc5ce6fdd118e4a437c9195127d449f5f9511de882b2868af2196d45683c8c3"}
Nov 03 03:52:33.391684 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.391526    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podStartSLOduration=2.391504972 podStartE2EDuration="2.391504972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:33.39095428 +0000 UTC m=+3044.621846699" watchObservedRunningTime="2026-11-03 03:52:33.391504972 +0000 UTC m=+3044.622397391"
Nov 03 03:52:33.438749 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.438712    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:33.444328 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.444300    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:33.790370 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.790340    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:33.952981 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:33.952944    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:34.103657 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:34.102060    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:34.241493 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:34.241454    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:34.242488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:34.242466    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:52:36.215841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:36.215807    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:36.439011 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:36.438979    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:36.589301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:36.589268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:39.280318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:39.280278    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:39.545246 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:39.545170    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:39.732586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:39.732523    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:40.120995 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.120961    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"]
Nov 03 03:52:40.123803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.123786    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.204577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.195493    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsqk\" (UniqueName: \"kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk\") pod \"payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.204577 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.195672    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.283575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.280794    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-7ed2d6-dockercfg-27qhz\""
Nov 03 03:52:40.283575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.281493    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-7ed2d6\""
Nov 03 03:52:40.324191 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.324153    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"]
Nov 03 03:52:40.324533 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.324499    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.324710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.324692    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsqk\" (UniqueName: \"kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk\") pod \"payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.326517 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.326235    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.329436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.329187    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"]
Nov 03 03:52:40.332924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.332903    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.347179 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.347151    5515 generic.go:356] "Generic (PLEG): container finished" podID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerID="3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7" exitCode=0
Nov 03 03:52:40.347311 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.347295    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" event={"ID":"cb8a6bb3-970e-4fce-bcad-01233bf889fc","Type":"ContainerDied","Data":"3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"}
Nov 03 03:52:40.347766 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.347749    5515 scope.go:122] "RemoveContainer" containerID="3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"
Nov 03 03:52:40.349459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.349432    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:40.349531 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.349497    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:40.349598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.349529    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:40.379360 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.379309    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"]
Nov 03 03:52:40.386868 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.386840    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsqk\" (UniqueName: \"kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk\") pod \"payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.436206 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.436181    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:40.537960 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.537931    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.538499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.538477    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92j8j\" (UniqueName: \"kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j\") pod \"payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.640599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.640478    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92j8j\" (UniqueName: \"kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j\") pod \"payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.640599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.640568    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.641527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.641502    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.703965 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.703929    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92j8j\" (UniqueName: \"kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j\") pod \"payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:40.826519 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:52:40.826488    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39620b2e_b2d5_4162_8115_d40e81939478.slice/crio-b64604a4cedf02c57530d2d276d4c8ca8af3a1a0e9b60f90cfafc3653d31d975 WatchSource:0}: Error finding container b64604a4cedf02c57530d2d276d4c8ca8af3a1a0e9b60f90cfafc3653d31d975: Status 404 returned error can't find the container with id b64604a4cedf02c57530d2d276d4c8ca8af3a1a0e9b60f90cfafc3653d31d975
Nov 03 03:52:40.826699 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.826675    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"]
Nov 03 03:52:40.952143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:40.952113    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:41.355039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.354948    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" event={"ID":"39620b2e-b2d5-4162-8115-d40e81939478","Type":"ContainerStarted","Data":"2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931"}
Nov 03 03:52:41.355039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.355008    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" event={"ID":"39620b2e-b2d5-4162-8115-d40e81939478","Type":"ContainerStarted","Data":"b64604a4cedf02c57530d2d276d4c8ca8af3a1a0e9b60f90cfafc3653d31d975"}
Nov 03 03:52:41.355622 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.355265    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:41.358919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.358894    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"]
Nov 03 03:52:41.360793 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:52:41.360772    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a63b964_d5a5_489d_becb_6c7fd95ec450.slice/crio-540f87e7dde213b3c322f715935cb4e9ef43a395bd63750160dd2c751cb6064b WatchSource:0}: Error finding container 540f87e7dde213b3c322f715935cb4e9ef43a395bd63750160dd2c751cb6064b: Status 404 returned error can't find the container with id 540f87e7dde213b3c322f715935cb4e9ef43a395bd63750160dd2c751cb6064b
Nov 03 03:52:41.361250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.361227    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" event={"ID":"cb8a6bb3-970e-4fce-bcad-01233bf889fc","Type":"ContainerStarted","Data":"b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635"}
Nov 03 03:52:41.491576 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.490021    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" podStartSLOduration=1.489996657 podStartE2EDuration="1.489996657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:41.473332433 +0000 UTC m=+3052.704224874" watchObservedRunningTime="2026-11-03 03:52:41.489996657 +0000 UTC m=+3052.720889076"
Nov 03 03:52:41.729586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.729520    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:52:41.731042 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.731017    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 03:52:41.731187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:41.731067    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 03:52:42.393610 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:42.393568    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" event={"ID":"3a63b964-d5a5-489d-becb-6c7fd95ec450","Type":"ContainerStarted","Data":"339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f"}
Nov 03 03:52:42.394187 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:42.393635    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" event={"ID":"3a63b964-d5a5-489d-becb-6c7fd95ec450","Type":"ContainerStarted","Data":"540f87e7dde213b3c322f715935cb4e9ef43a395bd63750160dd2c751cb6064b"}
Nov 03 03:52:42.394272 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:42.394213    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:42.513216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:42.513161    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" podStartSLOduration=2.513146082 podStartE2EDuration="2.513146082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:42.511308457 +0000 UTC m=+3053.742200874" watchObservedRunningTime="2026-11-03 03:52:42.513146082 +0000 UTC m=+3053.744038532"
Nov 03 03:52:42.729920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:42.729838    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 03:52:42.729920 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:42.729908    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 03:52:43.734584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:43.734072    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 03:52:43.734584 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:43.734134    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 03:52:44.591176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:44.591061    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:44.730066 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:44.730032    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 03:52:44.730245 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:44.730090    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 03:52:45.729890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:45.729858    5515 patch_prober.go:29] interesting pod/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 03:52:45.730323 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:45.729909    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 03:52:46.040067 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:46.039980    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:46.051926 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:46.051891    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:52:46.515004 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:46.514957    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:46.734309 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:46.734283    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:52:46.735614 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:46.735593    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 03:52:47.054816 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.054786    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"]
Nov 03 03:52:47.058118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.058086    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.138400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.138371    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9daccc-openshift-default-dockercfg-xt2zg\""
Nov 03 03:52:47.145613 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.145526    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:47.154448 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.154421    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"]
Nov 03 03:52:47.186566 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186507    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hmd\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.186800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186641    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.186800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186694    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.186800 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186751    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.186972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186835    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.186972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186885    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.186972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.186919    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.187108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.187012    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.187108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.187040    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.290183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.290143    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.293686    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.293727    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.293782    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hmd\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.293859    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.293907    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.293965    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.294061    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.294109    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.291010    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.295155    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.298574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.295379    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.313713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.299721    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.313964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.302709    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.314079 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.303392    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.343570 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.341067    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.362290 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.357513    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hmd\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.370668 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.370632    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token\") pod \"e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") " pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.374930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.374909    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:47.500582 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.497801    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:47.807417 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.807392    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"]
Nov 03 03:52:47.808271 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:52:47.808248    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce77fd3_d98f_4ac4_b8ba_562e561c8dff.slice/crio-71a3d5abf3e3740e2c035e8726992a4343efebc3e739cf1773c4c94f3b5d7e46 WatchSource:0}: Error finding container 71a3d5abf3e3740e2c035e8726992a4343efebc3e739cf1773c4c94f3b5d7e46: Status 404 returned error can't find the container with id 71a3d5abf3e3740e2c035e8726992a4343efebc3e739cf1773c4c94f3b5d7e46
Nov 03 03:52:47.810803 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.810770    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:47.810883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.810837    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:47.810883 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:47.810872    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:52:48.442413 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:48.442375    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" event={"ID":"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff","Type":"ContainerStarted","Data":"bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"}
Nov 03 03:52:48.442664 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:48.442430    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" event={"ID":"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff","Type":"ContainerStarted","Data":"71a3d5abf3e3740e2c035e8726992a4343efebc3e739cf1773c4c94f3b5d7e46"}
Nov 03 03:52:48.571353 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:48.571298    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podStartSLOduration=1.571284206 podStartE2EDuration="1.571284206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:52:48.570845361 +0000 UTC m=+3059.801737790" watchObservedRunningTime="2026-11-03 03:52:48.571284206 +0000 UTC m=+3059.802176621"
Nov 03 03:52:48.593767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:48.593735    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:52:49.380935 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:49.380902    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:49.398253 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:49.398196    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:49.449090 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:49.448985    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:49.449925 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:49.449900    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:52:49.871190 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:49.871158    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:50.994221 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:52:50.994173    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:52:51.380620 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:51.380583    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:51.558666 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:51.558634    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:51.702504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:51.702428    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:52.397643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:52.397615    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 03:52:54.409436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:54.409408    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 03:52:54.637001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:54.636974    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:54.841282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:54.841149    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:55.129298 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:55.129270    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:57.499322 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:57.499290    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:57.955782 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:57.955749    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:58.646212 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:58.638949    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:52:58.858652 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:52:58.858615    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:00.465257 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:00.465217    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:03.024408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:03.024368    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:03.181933 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:03.181898    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:05.429297 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:05.429268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:05.774499 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:05.774418    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:08.906880 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:08.906834    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:09.101581 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.101529    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:09.313859 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.313776    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"]
Nov 03 03:53:09.319884 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.319862    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.375748 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.375717    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"]
Nov 03 03:53:09.419589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.419542    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.419802 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.419782    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42hr\" (UniqueName: \"kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr\") pod \"payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.423359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.423311    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3d8dc5\""
Nov 03 03:53:09.423727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.423696    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3d8dc5-dockercfg-42mjx\""
Nov 03 03:53:09.512574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.510038    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"]
Nov 03 03:53:09.557571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.553902    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.557571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.553983    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42hr\" (UniqueName: \"kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr\") pod \"payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.557571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.555601    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.575702 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.574861    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.615162 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.615127    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"]
Nov 03 03:53:09.622107 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.622076    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42hr\" (UniqueName: \"kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr\") pod \"payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.633695 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.633672    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:09.655282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.655250    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.655409 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.655309    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj9b\" (UniqueName: \"kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b\") pod \"payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.756940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.756908    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.757124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.756951    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj9b\" (UniqueName: \"kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b\") pod \"payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.757644 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.757624    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.819379 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.819346    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj9b\" (UniqueName: \"kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b\") pod \"payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:09.919924 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:09.919893    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:10.010425 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.010395    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"]
Nov 03 03:53:10.012582 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:53:10.011696    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3de473_ca1b_4cdd_9696_d291633413b2.slice/crio-261ea5a7007da04d124a7c8e8c71cf644ace96fcf72349569b6c48fa9d9fee24 WatchSource:0}: Error finding container 261ea5a7007da04d124a7c8e8c71cf644ace96fcf72349569b6c48fa9d9fee24: Status 404 returned error can't find the container with id 261ea5a7007da04d124a7c8e8c71cf644ace96fcf72349569b6c48fa9d9fee24
Nov 03 03:53:10.297621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.297593    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"]
Nov 03 03:53:10.298543 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:53:10.298515    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1e6ce0_a826_4a6c_b6ac_75396e9825f9.slice/crio-752a4cb1d80ecb85a77315d9e4044df9d000affec1e872a6475b5e29ee85e4d4 WatchSource:0}: Error finding container 752a4cb1d80ecb85a77315d9e4044df9d000affec1e872a6475b5e29ee85e4d4: Status 404 returned error can't find the container with id 752a4cb1d80ecb85a77315d9e4044df9d000affec1e872a6475b5e29ee85e4d4
Nov 03 03:53:10.611161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.611129    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" event={"ID":"3a3de473-ca1b-4cdd-9696-d291633413b2","Type":"ContainerStarted","Data":"87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c"}
Nov 03 03:53:10.611368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.611190    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" event={"ID":"3a3de473-ca1b-4cdd-9696-d291633413b2","Type":"ContainerStarted","Data":"261ea5a7007da04d124a7c8e8c71cf644ace96fcf72349569b6c48fa9d9fee24"}
Nov 03 03:53:10.611368 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.611221    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:10.612783 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.612752    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" event={"ID":"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9","Type":"ContainerStarted","Data":"4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf"}
Nov 03 03:53:10.612930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.612833    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:10.612930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.612860    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" event={"ID":"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9","Type":"ContainerStarted","Data":"752a4cb1d80ecb85a77315d9e4044df9d000affec1e872a6475b5e29ee85e4d4"}
Nov 03 03:53:10.735451 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.735396    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" podStartSLOduration=1.735380804 podStartE2EDuration="1.735380804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:10.734214948 +0000 UTC m=+3081.965107374" watchObservedRunningTime="2026-11-03 03:53:10.735380804 +0000 UTC m=+3081.966273276"
Nov 03 03:53:10.771397 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:10.771365    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:11.144486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:11.144454    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:11.340728 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:11.340683    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:12.235672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:12.235642    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:12.434829 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:12.434002    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:16.555099 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:16.555063    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:17.098040 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:17.098010    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:17.354185 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:17.354109    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:17.512812 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:17.512782    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:20.384206 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.384174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:20.564713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.564682    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:20.804217 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.804106    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" podStartSLOduration=11.804090837 podStartE2EDuration="11.804090837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:10.868323282 +0000 UTC m=+3082.099215702" watchObservedRunningTime="2026-11-03 03:53:20.804090837 +0000 UTC m=+3092.034983282"
Nov 03 03:53:20.806974 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.806946    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"]
Nov 03 03:53:20.812365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.812345    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:20.869618 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.869584    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"]
Nov 03 03:53:20.945575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.941788    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-286a8c\""
Nov 03 03:53:20.958573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:20.955377    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-286a8c-dockercfg-xt69k\""
Nov 03 03:53:21.019727 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.019681    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl7r\" (UniqueName: \"kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r\") pod \"payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.020074 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.020055    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.055726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.051516    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"]
Nov 03 03:53:21.057139 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.057096    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"]
Nov 03 03:53:21.057337 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.057219    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.081112 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:53:21.081073    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:53:21.120907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.120872    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.121092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.121008    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.121092 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.121080    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl7r\" (UniqueName: \"kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r\") pod \"payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.121208 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.121186    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpjg\" (UniqueName: \"kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg\") pod \"payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.121615 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.121596    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.183408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.183375    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl7r\" (UniqueName: \"kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r\") pod \"payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.222018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.221981    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.222213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.222106    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpjg\" (UniqueName: \"kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg\") pod \"payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.222724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.222701    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.284744 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.284717    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpjg\" (UniqueName: \"kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg\") pod \"payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.377104 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.377073    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:21.426706 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.426666    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:21.623143 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.623116    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 03:53:21.623319 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.623187    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 03:53:21.758403 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.758371    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"]
Nov 03 03:53:21.761279 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:53:21.761250    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8434e7_5ef7_40f0_a981_cdbfb1e287b5.slice/crio-1ed86ee4342a283b138ee958073363653f0a8154c538c7a68f06b0cce1c43d8a WatchSource:0}: Error finding container 1ed86ee4342a283b138ee958073363653f0a8154c538c7a68f06b0cce1c43d8a: Status 404 returned error can't find the container with id 1ed86ee4342a283b138ee958073363653f0a8154c538c7a68f06b0cce1c43d8a
Nov 03 03:53:21.825118 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.825094    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:21.856770 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:21.856743    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"]
Nov 03 03:53:21.857468 ip-10-0-138-11 kubenswrapper[5515]: W1103 03:53:21.857439    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00600f9_826b_48ba_b576_dfe4e299957c.slice/crio-d3cd29d1d2b4c5a6f160d8a4ce6cde979fdef5e5416b4ee8e84015f494b1043e WatchSource:0}: Error finding container d3cd29d1d2b4c5a6f160d8a4ce6cde979fdef5e5416b4ee8e84015f494b1043e: Status 404 returned error can't find the container with id d3cd29d1d2b4c5a6f160d8a4ce6cde979fdef5e5416b4ee8e84015f494b1043e
Nov 03 03:53:22.762108 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.762074    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" event={"ID":"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5","Type":"ContainerStarted","Data":"b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73"}
Nov 03 03:53:22.762742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.762128    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" event={"ID":"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5","Type":"ContainerStarted","Data":"1ed86ee4342a283b138ee958073363653f0a8154c538c7a68f06b0cce1c43d8a"}
Nov 03 03:53:22.762742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.762172    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:22.763815 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.763790    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" event={"ID":"b00600f9-826b-48ba-b576-dfe4e299957c","Type":"ContainerStarted","Data":"e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6"}
Nov 03 03:53:22.763938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.763831    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" event={"ID":"b00600f9-826b-48ba-b576-dfe4e299957c","Type":"ContainerStarted","Data":"d3cd29d1d2b4c5a6f160d8a4ce6cde979fdef5e5416b4ee8e84015f494b1043e"}
Nov 03 03:53:22.763938 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.763862    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:22.880277 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:22.880220    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" podStartSLOduration=2.88020584 podStartE2EDuration="2.88020584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:22.879172008 +0000 UTC m=+3094.110064436" watchObservedRunningTime="2026-11-03 03:53:22.88020584 +0000 UTC m=+3094.111098255"
Nov 03 03:53:23.005586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:23.001245    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" podStartSLOduration=3.001222688 podStartE2EDuration="3.001222688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:53:22.997950593 +0000 UTC m=+3094.228843025" watchObservedRunningTime="2026-11-03 03:53:23.001222688 +0000 UTC m=+3094.232115109"
Nov 03 03:53:26.652453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:26.652419    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:31.465177 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:31.465144    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:33.773756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:33.773727    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 03:53:33.774228 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:33.773800    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 03:53:34.735536 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:34.735501    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:34.879760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:34.879727    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:36.202017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:36.201982    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:36.368587 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:36.368517    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:36.515621 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:36.515514    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:37.957725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:37.957692    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:41.655260 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:41.655226    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:42.164195 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:42.164139    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:46.630964 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:46.630930    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:50.985021 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:53:50.984972    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:53:51.368355 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:51.368324    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:51.529628 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:51.529592    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:51.675894 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:51.675808    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:53:55.400201 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:55.400168    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:53:56.379560 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:53:56.379512    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:01.343213 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:01.343178    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:05.282724 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:05.282690    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:54:05.283164 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:05.282757    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:54:05.786016 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:05.785976    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:06.061830 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.061737    5515 generic.go:356] "Generic (PLEG): container finished" podID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerID="b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb" exitCode=0
Nov 03 03:54:06.061996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.061817    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" event={"ID":"b9710500-d6bf-45c1-8724-8dcf08b046a5","Type":"ContainerDied","Data":"b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"}
Nov 03 03:54:06.062266 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.062252    5515 scope.go:122] "RemoveContainer" containerID="b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"
Nov 03 03:54:06.063838 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.063807    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:54:06.063971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.063889    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:54:06.063971 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.063938    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:54:06.236691 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:06.236659    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:07.092730 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:07.092697    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" event={"ID":"b9710500-d6bf-45c1-8724-8dcf08b046a5","Type":"ContainerStarted","Data":"0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a"}
Nov 03 03:54:07.438792 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:07.438755    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:54:07.439038 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:07.439016    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 03:54:07.439120 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:07.439101    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 03:54:08.439885 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:08.439851    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 03:54:08.440303 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:08.439913    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 03:54:09.439318 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:09.439281    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 03:54:09.439505 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:09.439339    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 03:54:10.439125 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:10.439087    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 03:54:10.439524 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:10.439142    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 03:54:10.947705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:10.947671    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:11.109073 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:11.109039    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:11.256243 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:11.256146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:11.439850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:11.439809    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 03:54:11.440300 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:11.439875    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 03:54:12.443334 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:12.443303    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:54:12.443784 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:12.443586    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:54:12.444369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:12.444349    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 03:54:15.986167 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:15.986123    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:20.985568 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:54:20.985515    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:54:22.417126 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:22.417046    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:22.563972 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:22.563935    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:25.578752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:25.578714    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:25.741496 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:25.741465    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:25.885608 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:25.885538    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:26.755336 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:26.755281    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:27.162459 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:27.162423    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:54:30.693275 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:30.693235    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:35.489716 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:35.489675    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:54:35.490252 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:35.489744    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:54:35.538647 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:35.538612    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:38.323124 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:38.323091    5515 generic.go:356] "Generic (PLEG): container finished" podID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerID="bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231" exitCode=0
Nov 03 03:54:38.323617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:38.323174    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" event={"ID":"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff","Type":"ContainerDied","Data":"bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"}
Nov 03 03:54:38.323617 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:38.323591    5515 scope.go:122] "RemoveContainer" containerID="bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"
Nov 03 03:54:38.325154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:38.325126    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:54:38.325227 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:38.325193    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:54:38.325269 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:38.325226    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 03:54:39.341619 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:39.341587    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" event={"ID":"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff","Type":"ContainerStarted","Data":"56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b"}
Nov 03 03:54:39.375875 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:39.375837    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:54:39.376262 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:39.376167    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 03 03:54:39.376439 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:39.376416    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 03 03:54:40.385176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:40.385116    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:40.386685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:40.386660    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 03 03:54:40.386921 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:40.386878    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 03 03:54:41.381934 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:41.381894    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 03 03:54:41.382250 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:41.382221    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 03 03:54:42.387001 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:42.386960    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 03 03:54:42.387436 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:42.387044    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 03 03:54:43.385711 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:43.385676    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 03 03:54:43.385990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:43.385964    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 03 03:54:44.382986 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:44.382946    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 03 03:54:44.383658 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:44.383624    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 03 03:54:44.780222 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:44.780190    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:45.193176 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:45.193138    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:45.379231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:45.379202    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:54:45.379721 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:45.379700    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:54:45.380498 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:45.380483    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 03:54:50.055314 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:50.055281    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:50.984940 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:54:50.984904    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:54:54.939850 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:54.939818    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:54:58.645168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:58.645134    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:54:59.832767 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:54:59.832734    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:00.033432 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:00.033370    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:03.087071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:03.087017    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"]
Nov 03 03:55:03.198606 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:03.198575    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"]
Nov 03 03:55:03.435930 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:03.435901    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:03.615940 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:03.615905    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:04.300128 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:04.300097    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:05.078013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:05.077977    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:06.126932 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:06.126902    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:08.088650 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.088585    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf" podUID="aacf57d0-664e-44ce-a75c-070023b37ac3" containerName="payload-processing" containerID="cri-o://0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a" gracePeriod=25
Nov 03 03:55:08.199737 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.199685    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z" podUID="cc837120-3a65-4b7a-99d9-82fe2665e12f" containerName="payload-pre-processing" containerID="cri-o://3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6" gracePeriod=25
Nov 03 03:55:08.378756 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.378726    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:55:08.474045 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.474020    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume\") pod \"aacf57d0-664e-44ce-a75c-070023b37ac3\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") "
Nov 03 03:55:08.474169 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.474101    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw\" (UniqueName: \"kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw\") pod \"aacf57d0-664e-44ce-a75c-070023b37ac3\" (UID: \"aacf57d0-664e-44ce-a75c-070023b37ac3\") "
Nov 03 03:55:08.474364 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.474344    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume" pod "aacf57d0-664e-44ce-a75c-070023b37ac3" (UID: "aacf57d0-664e-44ce-a75c-070023b37ac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:55:08.474739 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.474717    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacf57d0-664e-44ce-a75c-070023b37ac3-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:55:08.476559 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.476524    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw" pod "aacf57d0-664e-44ce-a75c-070023b37ac3" (UID: "aacf57d0-664e-44ce-a75c-070023b37ac3"). InnerVolumeSpecName "kube-api-access-q7blw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:55:08.494855 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.494832    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:55:08.575643 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.575616    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume\") pod \"cc837120-3a65-4b7a-99d9-82fe2665e12f\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") "
Nov 03 03:55:08.575814 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.575748    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s\" (UniqueName: \"kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s\") pod \"cc837120-3a65-4b7a-99d9-82fe2665e12f\" (UID: \"cc837120-3a65-4b7a-99d9-82fe2665e12f\") "
Nov 03 03:55:08.576096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.576070    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume" pod "cc837120-3a65-4b7a-99d9-82fe2665e12f" (UID: "cc837120-3a65-4b7a-99d9-82fe2665e12f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:55:08.576154 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.576115    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7blw\" (UniqueName: \"kubernetes.io/projected/aacf57d0-664e-44ce-a75c-070023b37ac3-kube-api-access-q7blw\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:55:08.577994 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.577973    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s" pod "cc837120-3a65-4b7a-99d9-82fe2665e12f" (UID: "cc837120-3a65-4b7a-99d9-82fe2665e12f"). InnerVolumeSpecName "kube-api-access-sn58s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:55:08.664707 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.664673    5515 generic.go:356] "Generic (PLEG): container finished" podID="cc837120-3a65-4b7a-99d9-82fe2665e12f" containerID="3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6" exitCode=0
Nov 03 03:55:08.664890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.664715    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"
Nov 03 03:55:08.664890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.664757    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z" event={"ID":"cc837120-3a65-4b7a-99d9-82fe2665e12f","Type":"ContainerDied","Data":"3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6"}
Nov 03 03:55:08.664890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.664839    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z" event={"ID":"cc837120-3a65-4b7a-99d9-82fe2665e12f","Type":"ContainerDied","Data":"49bbcf2a5ed90dece4acd64e47af573ef8386833ef25af2be151b397cf5ddfeb"}
Nov 03 03:55:08.664890 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.664864    5515 scope.go:122] "RemoveContainer" containerID="3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6"
Nov 03 03:55:08.666342 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.666318    5515 generic.go:356] "Generic (PLEG): container finished" podID="aacf57d0-664e-44ce-a75c-070023b37ac3" containerID="0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a" exitCode=0
Nov 03 03:55:08.666488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.666364    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"
Nov 03 03:55:08.666488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.666407    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf" event={"ID":"aacf57d0-664e-44ce-a75c-070023b37ac3","Type":"ContainerDied","Data":"0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a"}
Nov 03 03:55:08.666488 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.666460    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf" event={"ID":"aacf57d0-664e-44ce-a75c-070023b37ac3","Type":"ContainerDied","Data":"c65a579f7cd889e1a05c055e9bfbf691ab1714bbc5ad69a5062ab32f80958f13"}
Nov 03 03:55:08.678019 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.677996    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc837120-3a65-4b7a-99d9-82fe2665e12f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:55:08.678133 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.678020    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn58s\" (UniqueName: \"kubernetes.io/projected/cc837120-3a65-4b7a-99d9-82fe2665e12f-kube-api-access-sn58s\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:55:08.678629 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.678601    5515 scope.go:122] "RemoveContainer" containerID="3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6"
Nov 03 03:55:08.678878 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:55:08.678862    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6\": container with ID starting with 3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6 not found: ID does not exist" containerID="3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6"
Nov 03 03:55:08.678941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.678886    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6"} err="failed to get container status \"3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6\": rpc error: code = NotFound desc = could not find container \"3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6\": container with ID starting with 3b3355387ac63fed7823c9ef91d76dcfd012771cdb264f33611022144b42d8c6 not found: ID does not exist"
Nov 03 03:55:08.678941 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.678901    5515 scope.go:122] "RemoveContainer" containerID="0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a"
Nov 03 03:55:08.687907 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.687892    5515 scope.go:122] "RemoveContainer" containerID="0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a"
Nov 03 03:55:08.688210 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:55:08.688190    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a\": container with ID starting with 0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a not found: ID does not exist" containerID="0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a"
Nov 03 03:55:08.688268 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.688217    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a"} err="failed to get container status \"0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a\": rpc error: code = NotFound desc = could not find container \"0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a\": container with ID starting with 0018fd1dc814cbdbfc052cc10e4696c4d0d7e24b98a59ecee7a5c3db0ba5655a not found: ID does not exist"
Nov 03 03:55:08.833486 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.833445    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"]
Nov 03 03:55:08.837128 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.837105    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-158d23-859bc48879-lcf8z"]
Nov 03 03:55:08.894473 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.894449    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc837120-3a65-4b7a-99d9-82fe2665e12f" path="/var/lib/kubelet/pods/cc837120-3a65-4b7a-99d9-82fe2665e12f/volumes"
Nov 03 03:55:08.946836 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.946782    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"]
Nov 03 03:55:08.952183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:08.952164    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-158d23-5979cb56b7-t6xpf"]
Nov 03 03:55:10.150831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:10.150790    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:10.323690 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:10.323659    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:10.901599 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:10.901542    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacf57d0-664e-44ce-a75c-070023b37ac3" path="/var/lib/kubelet/pods/aacf57d0-664e-44ce-a75c-070023b37ac3/volumes"
Nov 03 03:55:14.962385 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:14.962351    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:15.608018 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:15.607950    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:20.517790 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:20.517755    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:20.985598 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:55:20.985543    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:55:25.324984 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:25.324948    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:30.167402 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:30.167364    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:31.738489 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:31.738454    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:35.452431 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:35.452380    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:40.337184 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:40.337139    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:45.225900 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:45.225846    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:46.788008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:46.787969    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:50.321410 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:50.321373    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:50.916390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:50.916339    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:55:51.009225 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:55:51.009189    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:55:55.456216 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:55:55.456128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:00.293630 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:00.293595    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:56:00.417131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:00.417102    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:05.338013 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:05.337981    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:05.986292 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:05.986253    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:06.494527 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:06.494497    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:10.230841 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:10.230808    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:15.077713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:15.077679    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:20.001705 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:20.001673    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:20.985418 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:56:20.985383    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:56:23.482635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:23.482594    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:24.865419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:24.865373    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:29.760155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:29.760113    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:34.592327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:34.592293    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:34.739047 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:34.739016    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:35.377919 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:35.377889    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:35.523586 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:35.523541    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:39.489985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:39.489931    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:44.471860 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:44.471821    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:48.185598 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:48.185559    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:56:48.571444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:48.571368    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:49.542135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:49.542101    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:50.170982 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:50.170950    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:56:50.173359 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:50.173327    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:56:50.205678 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:50.205656    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 03:56:50.207963 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:50.207945    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 03:56:50.985500 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:56:50.985463    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:56:53.009929 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:53.009897    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"]
Nov 03 03:56:53.010365 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:53.010097    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy" containerID="cri-o://ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79" gracePeriod=30
Nov 03 03:56:53.428869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:53.428840    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:54.392292 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:54.392256    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:56:58.292797 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.292726    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:56:58.444696 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.444660    5515 generic.go:356] "Generic (PLEG): container finished" podID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerID="ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79" exitCode=0
Nov 03 03:56:58.444871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.444705    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"
Nov 03 03:56:58.444871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.444746    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" event={"ID":"ed637db4-0648-4435-ad95-0fa224eeed5a","Type":"ContainerDied","Data":"ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79"}
Nov 03 03:56:58.444871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.444798    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7" event={"ID":"ed637db4-0648-4435-ad95-0fa224eeed5a","Type":"ContainerDied","Data":"43970f3fa838786634ce3ead5b419630cb09e0c063b8daf8976f4da1c23878cc"}
Nov 03 03:56:58.444871 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.444821    5515 scope.go:122] "RemoveContainer" containerID="ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79"
Nov 03 03:56:58.451186 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451158    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451250    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451301 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451292    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451457 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451423    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451475    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451574 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451510    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451631    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451671    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451708    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data\") pod \"ed637db4-0648-4435-ad95-0fa224eeed5a\" (UID: \"ed637db4-0648-4435-ad95-0fa224eeed5a\") "
Nov 03 03:56:58.451794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451742    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:56:58.451794 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451752    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:56:58.452231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.451918    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:56:58.452231 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.452141    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:56:58.452542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.452345    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:56:58.452542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.452505    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.452542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.452527    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed637db4-0648-4435-ad95-0fa224eeed5a-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.452542 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.452542    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.452865 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.452572    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.454287 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.454252    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:56:58.454419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.454394    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:56:58.454588 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.454482    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:56:58.454870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.454851    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg" pod "ed637db4-0648-4435-ad95-0fa224eeed5a" (UID: "ed637db4-0648-4435-ad95-0fa224eeed5a"). InnerVolumeSpecName "kube-api-access-zpprg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:56:58.456726 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.456707    5515 scope.go:122] "RemoveContainer" containerID="ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79"
Nov 03 03:56:58.457022 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:56:58.457001    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79\": container with ID starting with ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79 not found: ID does not exist" containerID="ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79"
Nov 03 03:56:58.457096 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.457033    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79"} err="failed to get container status \"ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79\": rpc error: code = NotFound desc = could not find container \"ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79\": container with ID starting with ad785c1598d8037a08486d619b8418380795f7a38cf49051e59b53913d6dcd79 not found: ID does not exist"
Nov 03 03:56:58.554369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.554277    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.554369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.554308    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpprg\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-kube-api-access-zpprg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.554369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.554324    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.554369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.554335    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.554369 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.554347    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed637db4-0648-4435-ad95-0fa224eeed5a-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:56:58.868928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.868901    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"]
Nov 03 03:56:58.921039 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:58.921003    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-qv7c7"]
Nov 03 03:56:59.229445 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:56:59.229330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:00.894870 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:00.894838    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" path="/var/lib/kubelet/pods/ed637db4-0648-4435-ad95-0fa224eeed5a/volumes"
Nov 03 03:57:04.195571 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:04.195521    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:06.850465 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:06.850432    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:08.389942 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:08.389907    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:10.122299 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:10.122268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:10.264415 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:10.264387    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:10.760944 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:10.760913    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:11.430797 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:11.430763    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:12.148573 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:12.146539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:12.396579 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:12.394595    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:14.119530 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:14.119499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:14.267182 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:14.267149    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:14.447443 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:14.447360    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:15.121428 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:15.121397    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:15.276327 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:15.276295    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:15.440046 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:15.439947    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:19.161419 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:19.161385    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"]
Nov 03 03:57:19.161973 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:19.161600    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" containerID="cri-o://49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291" gracePeriod=30
Nov 03 03:57:19.294453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:19.294423    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:20.362822 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:20.362791    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:20.985534 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:57:20.985496    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:57:24.428607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.428538    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:57:24.545156 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545122    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545166    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545238    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545332 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545292    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545349    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545468 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545380    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545765 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545433    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.545913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545676    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:57:24.545913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545687    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:57:24.545913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545761    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:57:24.545913 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.545903    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.546307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546191    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:57:24.546307 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546238    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert\") pod \"1b39e1f8-90c1-4954-865b-483226e9a0b0\" (UID: \"1b39e1f8-90c1-4954-865b-483226e9a0b0\") "
Nov 03 03:57:24.546595 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546570    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:57:24.546831 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546817    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.546910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546833    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b39e1f8-90c1-4954-865b-483226e9a0b0-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.546910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546843    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.546910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546852    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.546910 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.546860    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.547848 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.547830    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "kube-api-access-l7nhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:57:24.547967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.547846    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:57:24.548161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.548137    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:57:24.548161 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.548155    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo" pod "1b39e1f8-90c1-4954-865b-483226e9a0b0" (UID: "1b39e1f8-90c1-4954-865b-483226e9a0b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:57:24.648157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.648126    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nhn\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-kube-api-access-l7nhn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.648157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.648153    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.648157 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.648163    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.648400 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.648172    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b39e1f8-90c1-4954-865b-483226e9a0b0-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:57:24.675454 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.675427    5515 generic.go:356] "Generic (PLEG): container finished" podID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerID="49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291" exitCode=0
Nov 03 03:57:24.675635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.675518    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" event={"ID":"1b39e1f8-90c1-4954-865b-483226e9a0b0","Type":"ContainerDied","Data":"49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291"}
Nov 03 03:57:24.675635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.675530    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"
Nov 03 03:57:24.675635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.675601    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" event={"ID":"1b39e1f8-90c1-4954-865b-483226e9a0b0","Type":"ContainerDied","Data":"c64df5f3371af7af222f7327b2ba5b8a5d3c899698f5888c3f8393462b922d7b"}
Nov 03 03:57:24.675635 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.675624    5515 scope.go:122] "RemoveContainer" containerID="49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291"
Nov 03 03:57:24.688339 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.688318    5515 scope.go:122] "RemoveContainer" containerID="49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291"
Nov 03 03:57:24.688643 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:57:24.688621    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291\": container with ID starting with 49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291 not found: ID does not exist" containerID="49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291"
Nov 03 03:57:24.688718 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.688653    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291"} err="failed to get container status \"49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291\": rpc error: code = NotFound desc = could not find container \"49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291\": container with ID starting with 49f7c24c6659d3c5f038fdb64d86db19d47220ad6eddb298c6ebc78ab89e9291 not found: ID does not exist"
Nov 03 03:57:24.845065 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.845004    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"]
Nov 03 03:57:24.848363 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.848342    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66"]
Nov 03 03:57:24.896071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:24.896032    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" path="/var/lib/kubelet/pods/1b39e1f8-90c1-4954-865b-483226e9a0b0/volumes"
Nov 03 03:57:25.237444 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:25.237415    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:25.406135 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:25.406101    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:57:25.406315 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:25.406179    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-qjp66" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:57:27.979214 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:27.979172    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:57:30.133869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:30.133839    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:35.129967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:35.129921    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:40.008725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:40.008652    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:44.869000 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:44.868958    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:49.711985 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:49.711941    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:50.985477 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:57:50.985404    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:57:54.534330 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:54.534290    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:57:59.454928 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:57:59.454897    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:04.407541 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:04.407502    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:07.204398 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:07.204356    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:08.606679 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:08.606647    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"]
Nov 03 03:58:08.607202 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:08.606836    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy" containerID="cri-o://cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735" gracePeriod=30
Nov 03 03:58:09.215273 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:09.215237    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:09.685086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:09.685044    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:09.828641 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:09.828608    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:13.880950 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:13.880890    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:58:14.011747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011711    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.011747 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011750    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011787    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011818    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011875    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011963    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012012 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.011997    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012028    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012040    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:14.012291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012087    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert\") pod \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\" (UID: \"4b8cc06c-dc7b-4005-a69a-022fac0d2642\") "
Nov 03 03:58:14.012291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012146    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:14.012291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012156    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:14.012291 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012283    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:14.012760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012713    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:58:14.012760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012728    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.012760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012748    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.012760 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012761    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.012997 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.012775    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.014990 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.014963    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:58:14.015441 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.015413    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "kube-api-access-x9bd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:58:14.015589 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.015434    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:14.016575 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.016529    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo" pod "4b8cc06c-dc7b-4005-a69a-022fac0d2642" (UID: "4b8cc06c-dc7b-4005-a69a-022fac0d2642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:58:14.039967 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.039940    5515 generic.go:356] "Generic (PLEG): container finished" podID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerID="cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735" exitCode=0
Nov 03 03:58:14.040086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.039989    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" event={"ID":"4b8cc06c-dc7b-4005-a69a-022fac0d2642","Type":"ContainerDied","Data":"cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735"}
Nov 03 03:58:14.040086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.040001    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"
Nov 03 03:58:14.040086 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.040067    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw" event={"ID":"4b8cc06c-dc7b-4005-a69a-022fac0d2642","Type":"ContainerDied","Data":"9440e8950b0c78f49ad8f918cd5bd9e4c9468353e2f16813db6ec8cf77c55c6b"}
Nov 03 03:58:14.040209 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.040093    5515 scope.go:122] "RemoveContainer" containerID="cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735"
Nov 03 03:58:14.050453 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.050437    5515 scope.go:122] "RemoveContainer" containerID="cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735"
Nov 03 03:58:14.050746 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:58:14.050727    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735\": container with ID starting with cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735 not found: ID does not exist" containerID="cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735"
Nov 03 03:58:14.050815 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.050753    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735"} err="failed to get container status \"cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735\": rpc error: code = NotFound desc = could not find container \"cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735\": container with ID starting with cea1b9726feae82cd0f6e10c6d10eb2039b971d0150664cef167de1bd3ff9735 not found: ID does not exist"
Nov 03 03:58:14.058672 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.058638    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:14.114504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.114475    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.114504 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.114502    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-kube-api-access-x9bd8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.114713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.114514    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.114713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.114523    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.114713 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.114531    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8cc06c-dc7b-4005-a69a-022fac0d2642-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:14.197670 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.197145    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"]
Nov 03 03:58:14.207869 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.207823    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-wzsvw"]
Nov 03 03:58:14.895197 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:14.895168    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" path="/var/lib/kubelet/pods/4b8cc06c-dc7b-4005-a69a-022fac0d2642/volumes"
Nov 03 03:58:17.593072 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:17.593037    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:58:18.072313 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:18.072261    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:18.912271 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:18.912234    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:19.649646 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:19.649610    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"]
Nov 03 03:58:19.649826 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:19.649807    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" containerID="cri-o://e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c" gracePeriod=30
Nov 03 03:58:20.984823 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:58:20.984777    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:58:23.734710 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:23.734657    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:24.157742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:24.157634    5515 patch_prober.go:29] interesting pod/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:58:24.157742 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:24.157707    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:58:24.934749 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:24.934685    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:58:25.067051 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067023    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067085    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067122    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067226 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067206    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067228    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067265    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067297    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067408 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067364    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067432    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket\") pod \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\" (UID: \"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3\") "
Nov 03 03:58:25.067669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067520    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:25.067669 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067570    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:25.067842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067722    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:25.067842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.067785    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:58:25.068153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.068042    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:25.068153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.068097    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.068153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.068113    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.068153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.068122    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.068153 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.068132    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.069483 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.069459    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:58:25.069607 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.069460    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:58:25.070344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.070319    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:58:25.070418 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.070324    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh" pod "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" (UID: "79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3"). InnerVolumeSpecName "kube-api-access-qmbjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:58:25.114071 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.114039    5515 generic.go:356] "Generic (PLEG): container finished" podID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerID="e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c" exitCode=0
Nov 03 03:58:25.114242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.114090    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"
Nov 03 03:58:25.114242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.114126    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" event={"ID":"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3","Type":"ContainerDied","Data":"e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c"}
Nov 03 03:58:25.114242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.114197    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654" event={"ID":"79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3","Type":"ContainerDied","Data":"92b23774a5cbec1146b4274cefda6fe743f27f751165263fa5fe2e1dff02a900"}
Nov 03 03:58:25.114242 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.114227    5515 scope.go:122] "RemoveContainer" containerID="e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c"
Nov 03 03:58:25.127583 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.127540    5515 scope.go:122] "RemoveContainer" containerID="e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c"
Nov 03 03:58:25.127863 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:58:25.127847    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c\": container with ID starting with e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c not found: ID does not exist" containerID="e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c"
Nov 03 03:58:25.127922 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.127871    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c"} err="failed to get container status \"e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c\": rpc error: code = NotFound desc = could not find container \"e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c\": container with ID starting with e5ad8f9ca7311e9214360c81f3d35d85bfc19c5a3604cd9ce2a5156b43efcd8c not found: ID does not exist"
Nov 03 03:58:25.169204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.169178    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.169204 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.169200    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.169352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.169212    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbjh\" (UniqueName: \"kubernetes.io/projected/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-kube-api-access-qmbjh\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.169352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.169222    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.169352 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.169231    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 03:58:25.282842 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.282758    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"]
Nov 03 03:58:25.287626 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:25.287602    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-gz654"]
Nov 03 03:58:26.894787 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:26.894746    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" path="/var/lib/kubelet/pods/79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3/volumes"
Nov 03 03:58:28.539058 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:28.539019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:33.392423 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:33.392383    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:33.804752 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:33.804665    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:37.066954 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:37.066920    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:38.209183 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:38.209149    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:40.321131 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:40.321098    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:43.051198 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:43.051162    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:44.748008 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:44.747977    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:44.927282 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:44.927245    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:45.265372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:45.265340    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:45.465155 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:45.465118    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:47.946450 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:47.946418    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:50.520200 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:50.520167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:50.986790 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:58:50.986760    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:58:52.835999 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:52.835904    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:58:57.735095 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:58:57.735062    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:00.972475 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:00.972443    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 03:59:02.633372 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:02.633339    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:07.547685 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:07.547650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:07.697293 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:07.697249    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:12.422346 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:12.422311    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:16.490030 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:16.489998    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:17.248357 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:17.248327    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:20.984570 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:59:20.984520    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:59:22.064017 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:22.063982    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:26.891248 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:26.891213    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:27.819487 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:27.819453    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:31.707168 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:31.707134    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:36.537424 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:36.537362    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:38.026507 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:38.026473    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:38.656996 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:38.656958    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:38.909117 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:38.909024    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:41.439754 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:41.439721    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:42.710725 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:42.710687    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:42.882034 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:42.882005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:43.023344 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:43.023263    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:45.916396 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:45.916366    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:46.097304 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:46.097264    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:46.761532 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:46.761492    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:50.985979 ip-10-0-138-11 kubenswrapper[5515]: E1103 03:59:50.985941    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 03:59:51.651390 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:51.651345    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 03:59:56.566591 ip-10-0-138-11 kubenswrapper[5515]: I1103 03:59:56.566542    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:00.198247 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198210    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"]
Nov 03 04:00:00.198810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198796    5515 cpu_manager.go:414] "containerMap: removing container" podUID="aacf57d0-664e-44ce-a75c-070023b37ac3" containerName="payload-processing"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198811    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacf57d0-664e-44ce-a75c-070023b37ac3" containerName="payload-processing"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198826    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198832    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198847    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198853    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198864    5515 cpu_manager.go:414] "containerMap: removing container" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198869    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198897    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy"
Nov 03 04:00:00.198905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198902    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198911    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cc837120-3a65-4b7a-99d9-82fe2665e12f" containerName="payload-pre-processing"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198915    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc837120-3a65-4b7a-99d9-82fe2665e12f" containerName="payload-pre-processing"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.198994    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacf57d0-664e-44ce-a75c-070023b37ac3" containerName="payload-processing"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.199003    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed637db4-0648-4435-ad95-0fa224eeed5a" containerName="istio-proxy"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.199011    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8cc06c-dc7b-4005-a69a-022fac0d2642" containerName="istio-proxy"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.199018    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cc67dd-bb35-4e7a-be16-1e39dbdcb9f3" containerName="istio-proxy"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.199029    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b39e1f8-90c1-4954-865b-483226e9a0b0" containerName="istio-proxy"
Nov 03 04:00:00.199203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.199034    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc837120-3a65-4b7a-99d9-82fe2665e12f" containerName="payload-pre-processing"
Nov 03 04:00:00.202512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.202489    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.205208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.205179    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"]
Nov 03 04:00:00.209106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.209088    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"]
Nov 03 04:00:00.209208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.209198    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.213389 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.213371    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.248967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.248944    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"]
Nov 03 04:00:00.252649 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.252636    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.288856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.288831    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.289002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.288975    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xp5\" (UniqueName: \"kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.289002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.289000    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.289120 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.289084    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894640-nxxg4\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.289177 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.289152    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.289262 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.289238    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjxf\" (UniqueName: \"kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.289320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.289297    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lv2\" (UniqueName: \"kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.289421 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.289405    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z88z\" (UniqueName: \"kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z\") pod \"maas-api-key-cleanup-29894640-nxxg4\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.341253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.341229    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\""
Nov 03 04:00:00.383665 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.383624    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"]
Nov 03 04:00:00.386243 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.386219    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"]
Nov 03 04:00:00.389306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.389281    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"]
Nov 03 04:00:00.391013 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.390995    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xp5\" (UniqueName: \"kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.391119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391022    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.391119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391060    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894640-nxxg4\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.391234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391129    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.391234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391171    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjxf\" (UniqueName: \"kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.391234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391214    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lv2\" (UniqueName: \"kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.391407 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391313    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z88z\" (UniqueName: \"kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z\") pod \"maas-api-key-cleanup-29894640-nxxg4\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.391407 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391343    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.392136 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391946    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894640-nxxg4\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.392136 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.391969    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.392312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.392186    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.392312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.392213    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.399704 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.399679    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"]
Nov 03 04:00:00.454164 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.454101    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lv2\" (UniqueName: \"kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.454164 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.454127    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjxf\" (UniqueName: \"kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.454672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.454653    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z88z\" (UniqueName: \"kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z\") pod \"maas-api-key-cleanup-29894640-nxxg4\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.454765 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.454673    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xp5\" (UniqueName: \"kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.517703 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.517678    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:00.528263 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.528240    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:00.534482 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.534462    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:00.567393 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.567366    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:00.906393 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.906368    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"]
Nov 03 04:00:00.907160 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:00:00.907128    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681e2216_54a8_4f4b_8039_d31a89700c34.slice/crio-66d9ab64251bcb0bc6b754e75d3f463da5846a86fa54b675ad88d9108f46d206 WatchSource:0}: Error finding container 66d9ab64251bcb0bc6b754e75d3f463da5846a86fa54b675ad88d9108f46d206: Status 404 returned error can't find the container with id 66d9ab64251bcb0bc6b754e75d3f463da5846a86fa54b675ad88d9108f46d206
Nov 03 04:00:00.956364 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.956336    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"]
Nov 03 04:00:00.957071 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:00:00.957045    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35889be3_0692_43c1_bb58_a356476eda14.slice/crio-703b484f7a921094afb64c4cc287f4bbe616e31624cb249a7094846aaeee40c5 WatchSource:0}: Error finding container 703b484f7a921094afb64c4cc287f4bbe616e31624cb249a7094846aaeee40c5: Status 404 returned error can't find the container with id 703b484f7a921094afb64c4cc287f4bbe616e31624cb249a7094846aaeee40c5
Nov 03 04:00:00.958823 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:00:00.958769    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a470c84_858c_4529_8932_52e0ad0c7f1d.slice/crio-b3228e197055c6eb12d7b9d3ceb79c00bfbec8ffa8b47c2bae9aef13cc8bfbe5 WatchSource:0}: Error finding container b3228e197055c6eb12d7b9d3ceb79c00bfbec8ffa8b47c2bae9aef13cc8bfbe5: Status 404 returned error can't find the container with id b3228e197055c6eb12d7b9d3ceb79c00bfbec8ffa8b47c2bae9aef13cc8bfbe5
Nov 03 04:00:00.959372 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.959348    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"]
Nov 03 04:00:00.993779 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:00:00.993121    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd54476_e393_4907_9e54_901b9d27a50f.slice/crio-abd0235404366211f33951f40206d16f12e8f90eae2ed4bcee950d38b3e50ae8 WatchSource:0}: Error finding container abd0235404366211f33951f40206d16f12e8f90eae2ed4bcee950d38b3e50ae8: Status 404 returned error can't find the container with id abd0235404366211f33951f40206d16f12e8f90eae2ed4bcee950d38b3e50ae8
Nov 03 04:00:00.993779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:00.993428    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"]
Nov 03 04:00:01.444590 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.444543    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:01.720063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.720030    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerStarted","Data":"a7aa919731fe5c7b7f430ad550139f47ca0bd421f016998d3924c047cad8bfce"}
Nov 03 04:00:01.720295 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.720079    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerStarted","Data":"703b484f7a921094afb64c4cc287f4bbe616e31624cb249a7094846aaeee40c5"}
Nov 03 04:00:01.721896 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.721873    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerStarted","Data":"442fbf7e026a09da3bd7824f9711bb7a6a6f12cd1980e68417bde73404a9e16e"}
Nov 03 04:00:01.722031 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.721921    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerStarted","Data":"66d9ab64251bcb0bc6b754e75d3f463da5846a86fa54b675ad88d9108f46d206"}
Nov 03 04:00:01.723522 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.723500    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerStarted","Data":"5a7e7ab4b398097736e60b9ffd2806c89beedcd5e21b971269f0b72a6208455c"}
Nov 03 04:00:01.723654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.723559    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerStarted","Data":"abd0235404366211f33951f40206d16f12e8f90eae2ed4bcee950d38b3e50ae8"}
Nov 03 04:00:01.725192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.725163    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerStarted","Data":"f7564bad1aac24fda8f163d487ffda435d35e920921ca0785583cd09c9641a85"}
Nov 03 04:00:01.725278 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.725208    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerStarted","Data":"b3228e197055c6eb12d7b9d3ceb79c00bfbec8ffa8b47c2bae9aef13cc8bfbe5"}
Nov 03 04:00:01.839208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.839150    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" podStartSLOduration=1.839135985 podStartE2EDuration="1.839135985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:01.837977945 +0000 UTC m=+3493.068870367" watchObservedRunningTime="2026-11-03 04:00:01.839135985 +0000 UTC m=+3493.070028402"
Nov 03 04:00:01.956879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:01.956826    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" podStartSLOduration=1.956810261 podStartE2EDuration="1.956810261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:01.954982702 +0000 UTC m=+3493.185875119" watchObservedRunningTime="2026-11-03 04:00:01.956810261 +0000 UTC m=+3493.187702701"
Nov 03 04:00:02.075313 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.075265    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" podStartSLOduration=2.07525059 podStartE2EDuration="2.07525059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:02.073690626 +0000 UTC m=+3493.304583043" watchObservedRunningTime="2026-11-03 04:00:02.07525059 +0000 UTC m=+3493.306143024"
Nov 03 04:00:02.193681 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.193628    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" podStartSLOduration=2.193613961 podStartE2EDuration="2.193613961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:02.191194874 +0000 UTC m=+3493.422087292" watchObservedRunningTime="2026-11-03 04:00:02.193613961 +0000 UTC m=+3493.424506430"
Nov 03 04:00:02.732530 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.732496    5515 generic.go:356] "Generic (PLEG): container finished" podID="35889be3-0692-43c1-bb58-a356476eda14" containerID="a7aa919731fe5c7b7f430ad550139f47ca0bd421f016998d3924c047cad8bfce" exitCode=60
Nov 03 04:00:02.733006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.732598    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerDied","Data":"a7aa919731fe5c7b7f430ad550139f47ca0bd421f016998d3924c047cad8bfce"}
Nov 03 04:00:02.733006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.732835    5515 scope.go:122] "RemoveContainer" containerID="a7aa919731fe5c7b7f430ad550139f47ca0bd421f016998d3924c047cad8bfce"
Nov 03 04:00:02.734391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.734262    5515 generic.go:356] "Generic (PLEG): container finished" podID="681e2216-54a8-4f4b-8039-d31a89700c34" containerID="442fbf7e026a09da3bd7824f9711bb7a6a6f12cd1980e68417bde73404a9e16e" exitCode=60
Nov 03 04:00:02.734391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.734290    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerDied","Data":"442fbf7e026a09da3bd7824f9711bb7a6a6f12cd1980e68417bde73404a9e16e"}
Nov 03 04:00:02.734593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.734542    5515 scope.go:122] "RemoveContainer" containerID="442fbf7e026a09da3bd7824f9711bb7a6a6f12cd1980e68417bde73404a9e16e"
Nov 03 04:00:02.736065 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.735930    5515 generic.go:356] "Generic (PLEG): container finished" podID="0fd54476-e393-4907-9e54-901b9d27a50f" containerID="5a7e7ab4b398097736e60b9ffd2806c89beedcd5e21b971269f0b72a6208455c" exitCode=60
Nov 03 04:00:02.736065 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.736002    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerDied","Data":"5a7e7ab4b398097736e60b9ffd2806c89beedcd5e21b971269f0b72a6208455c"}
Nov 03 04:00:02.736241 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.736223    5515 scope.go:122] "RemoveContainer" containerID="5a7e7ab4b398097736e60b9ffd2806c89beedcd5e21b971269f0b72a6208455c"
Nov 03 04:00:02.737913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.737895    5515 generic.go:356] "Generic (PLEG): container finished" podID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerID="f7564bad1aac24fda8f163d487ffda435d35e920921ca0785583cd09c9641a85" exitCode=60
Nov 03 04:00:02.737979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.737930    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerDied","Data":"f7564bad1aac24fda8f163d487ffda435d35e920921ca0785583cd09c9641a85"}
Nov 03 04:00:02.738122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:02.738110    5515 scope.go:122] "RemoveContainer" containerID="f7564bad1aac24fda8f163d487ffda435d35e920921ca0785583cd09c9641a85"
Nov 03 04:00:03.750071 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.750033    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerStarted","Data":"6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904"}
Nov 03 04:00:03.752102 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.752071    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerStarted","Data":"a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0"}
Nov 03 04:00:03.753888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.753866    5515 generic.go:356] "Generic (PLEG): container finished" podID="0fd54476-e393-4907-9e54-901b9d27a50f" containerID="dcaab9ebc531d468ca67a6f64dac317ba0d432b5527467a9e74afb2cc46934d1" exitCode=60
Nov 03 04:00:03.753985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.753949    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerDied","Data":"dcaab9ebc531d468ca67a6f64dac317ba0d432b5527467a9e74afb2cc46934d1"}
Nov 03 04:00:03.754041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.754016    5515 scope.go:122] "RemoveContainer" containerID="5a7e7ab4b398097736e60b9ffd2806c89beedcd5e21b971269f0b72a6208455c"
Nov 03 04:00:03.754223 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.754205    5515 scope.go:122] "RemoveContainer" containerID="dcaab9ebc531d468ca67a6f64dac317ba0d432b5527467a9e74afb2cc46934d1"
Nov 03 04:00:03.754502 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:03.754482    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894640-nxxg4_odh-ai-gateway-infra(0fd54476-e393-4907-9e54-901b9d27a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" podUID="0fd54476-e393-4907-9e54-901b9d27a50f"
Nov 03 04:00:03.755904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:03.755882    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerStarted","Data":"75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec"}
Nov 03 04:00:04.763999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.763967    5515 generic.go:356] "Generic (PLEG): container finished" podID="681e2216-54a8-4f4b-8039-d31a89700c34" containerID="a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0" exitCode=60
Nov 03 04:00:04.764503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.764050    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerDied","Data":"a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0"}
Nov 03 04:00:04.764503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.764127    5515 scope.go:122] "RemoveContainer" containerID="442fbf7e026a09da3bd7824f9711bb7a6a6f12cd1980e68417bde73404a9e16e"
Nov 03 04:00:04.764503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.764329    5515 scope.go:122] "RemoveContainer" containerID="a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0"
Nov 03 04:00:04.764691 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:04.764646    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp_odh-ai-gateway-infra(681e2216-54a8-4f4b-8039-d31a89700c34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" podUID="681e2216-54a8-4f4b-8039-d31a89700c34"
Nov 03 04:00:04.766074 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.766054    5515 scope.go:122] "RemoveContainer" containerID="dcaab9ebc531d468ca67a6f64dac317ba0d432b5527467a9e74afb2cc46934d1"
Nov 03 04:00:04.766314 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:04.766294    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894640-nxxg4_odh-ai-gateway-infra(0fd54476-e393-4907-9e54-901b9d27a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" podUID="0fd54476-e393-4907-9e54-901b9d27a50f"
Nov 03 04:00:04.767616 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.767597    5515 generic.go:356] "Generic (PLEG): container finished" podID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerID="75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec" exitCode=60
Nov 03 04:00:04.767699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.767669    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerDied","Data":"75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec"}
Nov 03 04:00:04.767881 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.767867    5515 scope.go:122] "RemoveContainer" containerID="75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec"
Nov 03 04:00:04.768106 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:04.768082    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt_odh-ai-gateway-infra(2a470c84-858c-4529-8932-52e0ad0c7f1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d"
Nov 03 04:00:04.769599 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.769575    5515 generic.go:356] "Generic (PLEG): container finished" podID="35889be3-0692-43c1-bb58-a356476eda14" containerID="6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904" exitCode=60
Nov 03 04:00:04.769691 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.769604    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerDied","Data":"6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904"}
Nov 03 04:00:04.769826 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.769811    5515 scope.go:122] "RemoveContainer" containerID="6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904"
Nov 03 04:00:04.770012 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:04.769996    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k_odh-ai-gateway-infra(35889be3-0692-43c1-bb58-a356476eda14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" podUID="35889be3-0692-43c1-bb58-a356476eda14"
Nov 03 04:00:04.777385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.777365    5515 scope.go:122] "RemoveContainer" containerID="f7564bad1aac24fda8f163d487ffda435d35e920921ca0785583cd09c9641a85"
Nov 03 04:00:04.789973 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:04.789956    5515 scope.go:122] "RemoveContainer" containerID="a7aa919731fe5c7b7f430ad550139f47ca0bd421f016998d3924c047cad8bfce"
Nov 03 04:00:05.777302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:05.777271    5515 scope.go:122] "RemoveContainer" containerID="6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904"
Nov 03 04:00:05.777762 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:05.777533    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k_odh-ai-gateway-infra(35889be3-0692-43c1-bb58-a356476eda14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" podUID="35889be3-0692-43c1-bb58-a356476eda14"
Nov 03 04:00:05.778997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:05.778973    5515 scope.go:122] "RemoveContainer" containerID="a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0"
Nov 03 04:00:05.779217 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:05.779184    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp_odh-ai-gateway-infra(681e2216-54a8-4f4b-8039-d31a89700c34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" podUID="681e2216-54a8-4f4b-8039-d31a89700c34"
Nov 03 04:00:05.780610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:05.780591    5515 scope.go:122] "RemoveContainer" containerID="75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec"
Nov 03 04:00:05.780783 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:05.780766    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt_odh-ai-gateway-infra(2a470c84-858c-4529-8932-52e0ad0c7f1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d"
Nov 03 04:00:06.337611 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:06.337572    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:07.903259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:07.903226    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:11.227013 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:11.226971    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:15.886819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:15.886788    5515 scope.go:122] "RemoveContainer" containerID="75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec"
Nov 03 04:00:16.117040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:16.117008    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:16.852341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:16.852304    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerStarted","Data":"75d51dc0b0f0e879ee3b6a48a5a55c289637cf21849ec13bcd8bd04cfe1aa8c2"}
Nov 03 04:00:16.886824 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:16.886798    5515 scope.go:122] "RemoveContainer" containerID="a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0"
Nov 03 04:00:17.861426 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.861395    5515 generic.go:356] "Generic (PLEG): container finished" podID="681e2216-54a8-4f4b-8039-d31a89700c34" containerID="a9cda2e642c6e31e12505a9ae5e3d8071fdf42f3146f0299ae822e6a3dd57b41" exitCode=60
Nov 03 04:00:17.861979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.861472    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerDied","Data":"a9cda2e642c6e31e12505a9ae5e3d8071fdf42f3146f0299ae822e6a3dd57b41"}
Nov 03 04:00:17.861979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.861577    5515 scope.go:122] "RemoveContainer" containerID="a483c5163ecd280e8cafeb79c13672a923f60ea8388b1e5a4e3dd811118e5bc0"
Nov 03 04:00:17.861979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.861959    5515 scope.go:122] "RemoveContainer" containerID="a9cda2e642c6e31e12505a9ae5e3d8071fdf42f3146f0299ae822e6a3dd57b41"
Nov 03 04:00:17.862323 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:17.862297    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp_odh-ai-gateway-infra(681e2216-54a8-4f4b-8039-d31a89700c34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" podUID="681e2216-54a8-4f4b-8039-d31a89700c34"
Nov 03 04:00:17.863441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.863417    5515 generic.go:356] "Generic (PLEG): container finished" podID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerID="75d51dc0b0f0e879ee3b6a48a5a55c289637cf21849ec13bcd8bd04cfe1aa8c2" exitCode=60
Nov 03 04:00:17.863594 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.863481    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerDied","Data":"75d51dc0b0f0e879ee3b6a48a5a55c289637cf21849ec13bcd8bd04cfe1aa8c2"}
Nov 03 04:00:17.863881 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.863864    5515 scope.go:122] "RemoveContainer" containerID="75d51dc0b0f0e879ee3b6a48a5a55c289637cf21849ec13bcd8bd04cfe1aa8c2"
Nov 03 04:00:17.864170 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:17.864152    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt_odh-ai-gateway-infra(2a470c84-858c-4529-8932-52e0ad0c7f1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d"
Nov 03 04:00:17.874121 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.874102    5515 scope.go:122] "RemoveContainer" containerID="75bee2861fb4ac9abd611e44c42d1749c6c9a59a1f26f643e8fc4f1e5944c2ec"
Nov 03 04:00:17.886778 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.886757    5515 scope.go:122] "RemoveContainer" containerID="6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904"
Nov 03 04:00:17.886938 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.886923    5515 scope.go:122] "RemoveContainer" containerID="dcaab9ebc531d468ca67a6f64dac317ba0d432b5527467a9e74afb2cc46934d1"
Nov 03 04:00:17.986369 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:17.986336    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"]
Nov 03 04:00:18.875688 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.875657    5515 generic.go:356] "Generic (PLEG): container finished" podID="35889be3-0692-43c1-bb58-a356476eda14" containerID="51f3a065e7292a416c73c80a0cca863647124ac098478477b42788910551caca" exitCode=60
Nov 03 04:00:18.876095 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.875724    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerDied","Data":"51f3a065e7292a416c73c80a0cca863647124ac098478477b42788910551caca"}
Nov 03 04:00:18.876095 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.875821    5515 scope.go:122] "RemoveContainer" containerID="6cde0d961d1c5cec764e90d1bdf227c47aa73ab5218b66dc418825d43ec93904"
Nov 03 04:00:18.876240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.876222    5515 scope.go:122] "RemoveContainer" containerID="51f3a065e7292a416c73c80a0cca863647124ac098478477b42788910551caca"
Nov 03 04:00:18.876717 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:18.876692    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k_odh-ai-gateway-infra(35889be3-0692-43c1-bb58-a356476eda14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" podUID="35889be3-0692-43c1-bb58-a356476eda14"
Nov 03 04:00:18.884329 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.884294    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerStarted","Data":"5bd01eba6f0771eca75875719643974099f3177321a09958dadc63e423b75a29"}
Nov 03 04:00:18.884794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.884773    5515 scope.go:122] "RemoveContainer" containerID="5bd01eba6f0771eca75875719643974099f3177321a09958dadc63e423b75a29"
Nov 03 04:00:18.885178 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:18.885151    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894640-nxxg4_odh-ai-gateway-infra(0fd54476-e393-4907-9e54-901b9d27a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" podUID="0fd54476-e393-4907-9e54-901b9d27a50f"
Nov 03 04:00:18.993385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:18.993345    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:00:19.005748 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.005707    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"]
Nov 03 04:00:19.085923 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.085888    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:19.149678 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.149654    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca\") pod \"2a470c84-858c-4529-8932-52e0ad0c7f1d\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") "
Nov 03 04:00:19.149819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.149770    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2\" (UniqueName: \"kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2\") pod \"2a470c84-858c-4529-8932-52e0ad0c7f1d\" (UID: \"2a470c84-858c-4529-8932-52e0ad0c7f1d\") "
Nov 03 04:00:19.149995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.149975    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca" pod "2a470c84-858c-4529-8932-52e0ad0c7f1d" (UID: "2a470c84-858c-4529-8932-52e0ad0c7f1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:00:19.150255 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.150237    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a470c84-858c-4529-8932-52e0ad0c7f1d-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:19.152224 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.152199    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2" pod "2a470c84-858c-4529-8932-52e0ad0c7f1d" (UID: "2a470c84-858c-4529-8932-52e0ad0c7f1d"). InnerVolumeSpecName "kube-api-access-n8lv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:00:19.167897 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.167875    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:19.251199 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.251168    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf\" (UniqueName: \"kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf\") pod \"681e2216-54a8-4f4b-8039-d31a89700c34\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") "
Nov 03 04:00:19.251396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.251220    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca\") pod \"681e2216-54a8-4f4b-8039-d31a89700c34\" (UID: \"681e2216-54a8-4f4b-8039-d31a89700c34\") "
Nov 03 04:00:19.251656 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.251631    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca" pod "681e2216-54a8-4f4b-8039-d31a89700c34" (UID: "681e2216-54a8-4f4b-8039-d31a89700c34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:00:19.251656 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.251650    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8lv2\" (UniqueName: \"kubernetes.io/projected/2a470c84-858c-4529-8932-52e0ad0c7f1d-kube-api-access-n8lv2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:19.253637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.253613    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf" pod "681e2216-54a8-4f4b-8039-d31a89700c34" (UID: "681e2216-54a8-4f4b-8039-d31a89700c34"). InnerVolumeSpecName "kube-api-access-fmjxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:00:19.375417 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.368936    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmjxf\" (UniqueName: \"kubernetes.io/projected/681e2216-54a8-4f4b-8039-d31a89700c34-kube-api-access-fmjxf\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:19.375417 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.368977    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681e2216-54a8-4f4b-8039-d31a89700c34-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:19.894592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.894534    5515 generic.go:356] "Generic (PLEG): container finished" podID="0fd54476-e393-4907-9e54-901b9d27a50f" containerID="5bd01eba6f0771eca75875719643974099f3177321a09958dadc63e423b75a29" exitCode=60
Nov 03 04:00:19.895057 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.894621    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerDied","Data":"5bd01eba6f0771eca75875719643974099f3177321a09958dadc63e423b75a29"}
Nov 03 04:00:19.895057 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.894702    5515 scope.go:122] "RemoveContainer" containerID="dcaab9ebc531d468ca67a6f64dac317ba0d432b5527467a9e74afb2cc46934d1"
Nov 03 04:00:19.895191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.895146    5515 scope.go:122] "RemoveContainer" containerID="5bd01eba6f0771eca75875719643974099f3177321a09958dadc63e423b75a29"
Nov 03 04:00:19.895487 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:19.895465    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894640-nxxg4_odh-ai-gateway-infra(0fd54476-e393-4907-9e54-901b9d27a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" podUID="0fd54476-e393-4907-9e54-901b9d27a50f"
Nov 03 04:00:19.896214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.896193    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"
Nov 03 04:00:19.896356 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.896304    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt" event={"ID":"2a470c84-858c-4529-8932-52e0ad0c7f1d","Type":"ContainerDied","Data":"b3228e197055c6eb12d7b9d3ceb79c00bfbec8ffa8b47c2bae9aef13cc8bfbe5"}
Nov 03 04:00:19.899208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.899180    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp" event={"ID":"681e2216-54a8-4f4b-8039-d31a89700c34","Type":"ContainerDied","Data":"66d9ab64251bcb0bc6b754e75d3f463da5846a86fa54b675ad88d9108f46d206"}
Nov 03 04:00:19.899327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.899260    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"
Nov 03 04:00:19.908857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.908837    5515 scope.go:122] "RemoveContainer" containerID="75d51dc0b0f0e879ee3b6a48a5a55c289637cf21849ec13bcd8bd04cfe1aa8c2"
Nov 03 04:00:19.917621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:19.917603    5515 scope.go:122] "RemoveContainer" containerID="a9cda2e642c6e31e12505a9ae5e3d8071fdf42f3146f0299ae822e6a3dd57b41"
Nov 03 04:00:20.006708 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.006673    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"]
Nov 03 04:00:20.128286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.128255    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"]
Nov 03 04:00:20.139465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.139437    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3d8dc5-29894640-zwszt"]
Nov 03 04:00:20.157367 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.157336    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:20.178848 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.178821    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5\" (UniqueName: \"kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5\") pod \"35889be3-0692-43c1-bb58-a356476eda14\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") "
Nov 03 04:00:20.178969 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.178944    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca\") pod \"35889be3-0692-43c1-bb58-a356476eda14\" (UID: \"35889be3-0692-43c1-bb58-a356476eda14\") "
Nov 03 04:00:20.179270 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.179242    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca" pod "35889be3-0692-43c1-bb58-a356476eda14" (UID: "35889be3-0692-43c1-bb58-a356476eda14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:00:20.179408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.179396    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35889be3-0692-43c1-bb58-a356476eda14-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:20.181049 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.181028    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5" pod "35889be3-0692-43c1-bb58-a356476eda14" (UID: "35889be3-0692-43c1-bb58-a356476eda14"). InnerVolumeSpecName "kube-api-access-q8xp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:00:20.241428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.241395    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"]
Nov 03 04:00:20.280458 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.280423    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xp5\" (UniqueName: \"kubernetes.io/projected/35889be3-0692-43c1-bb58-a356476eda14-kube-api-access-q8xp5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:20.367131 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.367103    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"]
Nov 03 04:00:20.373971 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.373948    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894640-fgxvp"]
Nov 03 04:00:20.896866 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.896826    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" path="/var/lib/kubelet/pods/2a470c84-858c-4529-8932-52e0ad0c7f1d/volumes"
Nov 03 04:00:20.897477 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.897455    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" path="/var/lib/kubelet/pods/681e2216-54a8-4f4b-8039-d31a89700c34/volumes"
Nov 03 04:00:20.907065 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.907025    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k" event={"ID":"35889be3-0692-43c1-bb58-a356476eda14","Type":"ContainerDied","Data":"703b484f7a921094afb64c4cc287f4bbe616e31624cb249a7094846aaeee40c5"}
Nov 03 04:00:20.907260 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.907097    5515 scope.go:122] "RemoveContainer" containerID="51f3a065e7292a416c73c80a0cca863647124ac098478477b42788910551caca"
Nov 03 04:00:20.907260 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:20.907186    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"
Nov 03 04:00:20.996065 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:20.994879    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:00:21.048524 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.048487    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:21.077772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.077741    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"]
Nov 03 04:00:21.080924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.080821    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894640-g744k"]
Nov 03 04:00:21.080924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.080897    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:21.106281 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.105897    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca\") pod \"0fd54476-e393-4907-9e54-901b9d27a50f\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") "
Nov 03 04:00:21.106281 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.105957    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z\" (UniqueName: \"kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z\") pod \"0fd54476-e393-4907-9e54-901b9d27a50f\" (UID: \"0fd54476-e393-4907-9e54-901b9d27a50f\") "
Nov 03 04:00:21.130572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.128048    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca" pod "0fd54476-e393-4907-9e54-901b9d27a50f" (UID: "0fd54476-e393-4907-9e54-901b9d27a50f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:00:21.130572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.128514    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd54476-e393-4907-9e54-901b9d27a50f-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:21.145138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.145107    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z" pod "0fd54476-e393-4907-9e54-901b9d27a50f" (UID: "0fd54476-e393-4907-9e54-901b9d27a50f"). InnerVolumeSpecName "kube-api-access-2z88z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:00:21.229628 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.229589    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z88z\" (UniqueName: \"kubernetes.io/projected/0fd54476-e393-4907-9e54-901b9d27a50f-kube-api-access-2z88z\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:21.921053 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.921025    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"
Nov 03 04:00:21.921526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.921072    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4" event={"ID":"0fd54476-e393-4907-9e54-901b9d27a50f","Type":"ContainerDied","Data":"abd0235404366211f33951f40206d16f12e8f90eae2ed4bcee950d38b3e50ae8"}
Nov 03 04:00:21.921526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:21.921135    5515 scope.go:122] "RemoveContainer" containerID="5bd01eba6f0771eca75875719643974099f3177321a09958dadc63e423b75a29"
Nov 03 04:00:22.089240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:22.089204    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"]
Nov 03 04:00:22.095796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:22.095772    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894640-nxxg4"]
Nov 03 04:00:22.894533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:22.894461    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" path="/var/lib/kubelet/pods/0fd54476-e393-4907-9e54-901b9d27a50f/volumes"
Nov 03 04:00:22.894923 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:22.894907    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35889be3-0692-43c1-bb58-a356476eda14" path="/var/lib/kubelet/pods/35889be3-0692-43c1-bb58-a356476eda14/volumes"
Nov 03 04:00:24.650476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:24.650445    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"]
Nov 03 04:00:24.650926 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:24.650672    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy" containerID="cri-o://87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e" gracePeriod=30
Nov 03 04:00:25.930254 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:25.930204    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:29.919597 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.919529    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 04:00:29.973958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.973923    5515 generic.go:356] "Generic (PLEG): container finished" podID="a4f5990c-30ac-495d-851a-a865a50e5286" containerID="87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e" exitCode=0
Nov 03 04:00:29.973958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.973946    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"
Nov 03 04:00:29.974191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.973975    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" event={"ID":"a4f5990c-30ac-495d-851a-a865a50e5286","Type":"ContainerDied","Data":"87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e"}
Nov 03 04:00:29.974191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.974025    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2" event={"ID":"a4f5990c-30ac-495d-851a-a865a50e5286","Type":"ContainerDied","Data":"af8b41fafd8eb3a3869281cf6a4c5800ef9c0cedf14605c55366158fdf9c95df"}
Nov 03 04:00:29.974191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.974047    5515 scope.go:122] "RemoveContainer" containerID="87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e"
Nov 03 04:00:29.984510 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.984481    5515 scope.go:122] "RemoveContainer" containerID="87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e"
Nov 03 04:00:29.984813 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:29.984786    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e\": container with ID starting with 87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e not found: ID does not exist" containerID="87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e"
Nov 03 04:00:29.984881 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:29.984814    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e"} err="failed to get container status \"87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e\": rpc error: code = NotFound desc = could not find container \"87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e\": container with ID starting with 87f8092e535cfe565e335fb662b5a4d4ff13d7756449220579aca0cfb6dc4c8e not found: ID does not exist"
Nov 03 04:00:30.045642 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045615    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.045819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045686    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.045819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045736    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.045819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045778    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.045990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045905    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.045990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045913    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:00:30.045990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.045949    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.046141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046000    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.046141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046030    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.046141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046093    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket\") pod \"a4f5990c-30ac-495d-851a-a865a50e5286\" (UID: \"a4f5990c-30ac-495d-851a-a865a50e5286\") "
Nov 03 04:00:30.046141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046114    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:00:30.046302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046149    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:00:30.046302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046187    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:00:30.046461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.046435    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:00:30.047055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.047025    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4f5990c-30ac-495d-851a-a865a50e5286-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.047055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.047058    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.047284 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.047074    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.047284 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.047091    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.047284 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.047105    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.048779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.048754    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:00:30.048779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.048764    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:00:30.049008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.048987    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:00:30.049141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.049120    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp" pod "a4f5990c-30ac-495d-851a-a865a50e5286" (UID: "a4f5990c-30ac-495d-851a-a865a50e5286"). InnerVolumeSpecName "kube-api-access-rkwlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:00:30.148461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.148428    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4f5990c-30ac-495d-851a-a865a50e5286-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.148461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.148462    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4f5990c-30ac-495d-851a-a865a50e5286-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.148772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.148477    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.148772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.148491    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkwlp\" (UniqueName: \"kubernetes.io/projected/a4f5990c-30ac-495d-851a-a865a50e5286-kube-api-access-rkwlp\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:00:30.398035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.398002    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"]
Nov 03 04:00:30.403384 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.403356    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-sdrv2"]
Nov 03 04:00:30.758633 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.758597    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:30.894222 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:30.894186    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" path="/var/lib/kubelet/pods/a4f5990c-30ac-495d-851a-a865a50e5286/volumes"
Nov 03 04:00:35.598791 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:35.598750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:39.676669 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:39.676638    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:40.448794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:40.448758    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:45.271251 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:45.271205    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:49.803158 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803123    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"]
Nov 03 04:00:49.803825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803796    5515 cpu_manager.go:414] "containerMap: removing container" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:00:49.803825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803827    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803840    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803849    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803865    5515 cpu_manager.go:414] "containerMap: removing container" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803870    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803878    5515 cpu_manager.go:414] "containerMap: removing container" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803883    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803891    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803894    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803902    5515 cpu_manager.go:414] "containerMap: removing container" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803907    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803920    5515 cpu_manager.go:414] "containerMap: removing container" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:00:49.803927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803925    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803941    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803949    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803968    5515 cpu_manager.go:414] "containerMap: removing container" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.803973    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804076    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804092    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804103    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804110    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804115    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804124    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f5990c-30ac-495d-851a-a865a50e5286" containerName="istio-proxy"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804129    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804135    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:00:49.804360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.804140    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:00:49.807415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.807398    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:49.909711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.909671    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-5172b1-serving-cert\""
Nov 03 04:00:49.912435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.912411    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"]
Nov 03 04:00:49.918309 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.918288    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:49.918490 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.918472    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:49.918535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.918524    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2djgg\" (UniqueName: \"kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:49.918670 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:49.918647    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.001619 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.001586    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:50.020391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.020361    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.020602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.020414    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2djgg\" (UniqueName: \"kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.020602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.020440    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.020602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.020501    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.020730 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:50.020646    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-serving-cert: secret "maas-api-e2e-shared-a-w3-5172b1-serving-cert" not found
Nov 03 04:00:50.020772 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:50.020760    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls podName:3a08833e-102e-48b1-a16a-54de75ea98e9 nodeName:}" failed. No retries permitted until 2026-11-03 04:00:50.520736366 +0000 UTC m=+3541.751628765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls") pod "maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" (UID: "3a08833e-102e-48b1-a16a-54de75ea98e9") : secret "maas-api-e2e-shared-a-w3-5172b1-serving-cert" not found
Nov 03 04:00:50.021195 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.021173    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.023132 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.023110    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.086216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.086132    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2djgg\" (UniqueName: \"kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.164672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.164625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:50.314133 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.314102    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:50.529361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.529326    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.532114 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.532086    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.721495 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:50.721462    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:50.985871 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:00:50.985830    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:00:51.103748 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:51.103721    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"]
Nov 03 04:00:51.104805 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:00:51.104778    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a08833e_102e_48b1_a16a_54de75ea98e9.slice/crio-87c917188d9c7cf1582d46387b1badb4bf86a84cb78d4c2e696bd89c402897a4 WatchSource:0}: Error finding container 87c917188d9c7cf1582d46387b1badb4bf86a84cb78d4c2e696bd89c402897a4: Status 404 returned error can't find the container with id 87c917188d9c7cf1582d46387b1badb4bf86a84cb78d4c2e696bd89c402897a4
Nov 03 04:00:51.106372 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:51.106350    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:00:51.138948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:51.138915    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" event={"ID":"3a08833e-102e-48b1-a16a-54de75ea98e9","Type":"ContainerStarted","Data":"87c917188d9c7cf1582d46387b1badb4bf86a84cb78d4c2e696bd89c402897a4"}
Nov 03 04:00:53.152854 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:53.152820    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" event={"ID":"3a08833e-102e-48b1-a16a-54de75ea98e9","Type":"ContainerStarted","Data":"82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447"}
Nov 03 04:00:53.153342 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:53.152890    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:53.275976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:53.275918    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" podStartSLOduration=3.141898217 podStartE2EDuration="4.275898492s" totalImagesPullingTime="1.134000275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:00:53.273158888 +0000 UTC m=+3544.504051305" watchObservedRunningTime="2026-11-03 04:00:53.275898492 +0000 UTC m=+3544.506790910"
Nov 03 04:00:55.011151 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:55.011096    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:00:59.164951 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:59.164923    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:00:59.937435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:00:59.937398    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:03.817385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:03.817352    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:04.680755 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:04.680720    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:08.258817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:08.258782    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:09.570029 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:09.569995    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:14.069237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:14.069199    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:01:14.469591 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:14.469517    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:19.327749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:19.327701    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:20.985277 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:01:20.985238    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:01:24.267157 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:24.267118    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:29.038114 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:29.038081    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:33.923011 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:33.922973    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:38.817465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:38.817429    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:43.739198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:43.739162    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:48.638307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:48.638268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:50.211458 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:50.211432    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:01:50.213887 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:50.213860    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:01:50.246932 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:50.246905    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:01:50.249475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:50.249458    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:01:50.983717 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:01:50.983682    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:01:53.517959 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:53.517872    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:01:58.369266 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:01:58.369232    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:03.200880 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:03.200844    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:04.332824 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:04.332788    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:02:08.128828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:08.128795    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:08.608091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:08.608061    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:11.220136 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:11.220096    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:11.706221 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:11.706190    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:12.362425 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:12.362385    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:12.572863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:12.572833    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:13.250144 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:13.250116    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:14.728822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:14.728793    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:14.870482 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:14.870448    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:17.525929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:17.525896    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:17.693769 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:17.693735    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:17.837155 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:17.837064    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:18.461173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:18.461136    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:20.821618 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:20.821588    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:21.005000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:21.004952    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:21.035874 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:02:21.035835    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:02:21.666498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:21.666433    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:23.568664 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:23.568607    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:28.427396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:28.427357    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:33.327308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:33.327269    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:36.396516 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:36.396478    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:37.360725 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:37.360672    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:38.216352 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:38.216319    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:38.378409 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:38.378279    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:39.253449 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:39.253404    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:40.674308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:40.674271    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"]
Nov 03 04:02:40.674778 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:40.674518    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy" containerID="cri-o://2effd1b0fda5abe5b1880e5842bdbc49a519684690f4ed5e10cfdfa69d3ae1bf" gracePeriod=30
Nov 03 04:02:40.835707 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:40.835670    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:42.147090 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:42.147056    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:42.332684 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:42.332613    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:43.197958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:43.197924    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:45.881047 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:45.881015    5515 generic.go:356] "Generic (PLEG): container finished" podID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerID="2effd1b0fda5abe5b1880e5842bdbc49a519684690f4ed5e10cfdfa69d3ae1bf" exitCode=0
Nov 03 04:02:45.881929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:45.881096    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" event={"ID":"eff81363-5ff0-433f-b8b2-132e93a2765a","Type":"ContainerDied","Data":"2effd1b0fda5abe5b1880e5842bdbc49a519684690f4ed5e10cfdfa69d3ae1bf"}
Nov 03 04:02:45.949954 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:45.949898    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 04:02:46.026045 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026015    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026057    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026105    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026208    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026256    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026335    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026383    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026414    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026418    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:02:46.026652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026449    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket\") pod \"eff81363-5ff0-433f-b8b2-132e93a2765a\" (UID: \"eff81363-5ff0-433f-b8b2-132e93a2765a\") "
Nov 03 04:02:46.026652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026598    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:02:46.026766 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026751    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:02:46.026858 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.026827    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:02:46.027035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.027008    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:02:46.027260 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.027233    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff81363-5ff0-433f-b8b2-132e93a2765a-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.027347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.027260    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.027347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.027279    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.027347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.027296    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.027347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.027311    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.028652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.028624    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:02:46.029114 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.029095    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:02:46.029184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.029106    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:02:46.029184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.029133    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq" pod "eff81363-5ff0-433f-b8b2-132e93a2765a" (UID: "eff81363-5ff0-433f-b8b2-132e93a2765a"). InnerVolumeSpecName "kube-api-access-t78kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:02:46.110949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.110919    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:46.128542 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.128512    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.128542 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.128541    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.128542 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.128571    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.128781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.128585    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78kq\" (UniqueName: \"kubernetes.io/projected/eff81363-5ff0-433f-b8b2-132e93a2765a-kube-api-access-t78kq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:02:46.334946 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.334847    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:46.524727 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.524694    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:46.667492 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.667454    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:46.888821 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.888794    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"
Nov 03 04:02:46.894514 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.894489    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45" event={"ID":"eff81363-5ff0-433f-b8b2-132e93a2765a","Type":"ContainerDied","Data":"beadb6f3d0b53f9386925840017e9bb0b85c50c386b77744be83caf029e650fe"}
Nov 03 04:02:46.894639 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:46.894535    5515 scope.go:122] "RemoveContainer" containerID="2effd1b0fda5abe5b1880e5842bdbc49a519684690f4ed5e10cfdfa69d3ae1bf"
Nov 03 04:02:47.079022 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:47.078949    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"]
Nov 03 04:02:47.130706 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:47.130678    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b1ad4e-openshift-default-74bc5ff7f5-w8h45"]
Nov 03 04:02:47.649415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:47.649383    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:48.346304 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:48.346274    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:48.512216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:48.512181    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:48.654195 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:48.654160    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:48.904370 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:48.904284    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" path="/var/lib/kubelet/pods/eff81363-5ff0-433f-b8b2-132e93a2765a/volumes"
Nov 03 04:02:50.984970 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:02:50.984928    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:02:53.464601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:53.464539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:53.608640 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:53.608609    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:53.808528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:53.808419    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:53.952359 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:53.952321    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:54.038531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.038350    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"]
Nov 03 04:02:54.041609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041565    5515 cpu_manager.go:414] "containerMap: removing container" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy"
Nov 03 04:02:54.041609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041593    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041631    5515 cpu_manager.go:414] "containerMap: removing container" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041638    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041648    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041655    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041665    5515 cpu_manager.go:414] "containerMap: removing container" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041671    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041690    5515 cpu_manager.go:414] "containerMap: removing container" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:02:54.041844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041696    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:02:54.042328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041864    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd54476-e393-4907-9e54-901b9d27a50f" containerName="cleanup"
Nov 03 04:02:54.042328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041883    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a470c84-858c-4529-8932-52e0ad0c7f1d" containerName="cleanup"
Nov 03 04:02:54.042328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041896    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="35889be3-0692-43c1-bb58-a356476eda14" containerName="cleanup"
Nov 03 04:02:54.042328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041906    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff81363-5ff0-433f-b8b2-132e93a2765a" containerName="istio-proxy"
Nov 03 04:02:54.042328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.041923    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="681e2216-54a8-4f4b-8039-d31a89700c34" containerName="cleanup"
Nov 03 04:02:54.049589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.049525    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.094046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.094019    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"]
Nov 03 04:02:54.103607 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.103575    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:54.128273 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.128249    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.128437 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.128414    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9qx\" (UniqueName: \"kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx\") pod \"payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.141921 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.141895    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8c832d\""
Nov 03 04:02:54.142060 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.141949    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8c832d-dockercfg-xz77b\""
Nov 03 04:02:54.230027 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.230001    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.230173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.230094    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9qx\" (UniqueName: \"kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx\") pod \"payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.230825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.230805    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.244288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.244264    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:54.291945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.291918    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9qx\" (UniqueName: \"kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx\") pod \"payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.365697 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.365670    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:54.741438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.741405    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"]
Nov 03 04:02:54.742739 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:02:54.742700    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870e5933_2258_43e5_900d_2617a3a9650d.slice/crio-f3beb9ebd40ca1c38f6f897f6eb6eb878c0b22a9d72592e9a6a3f50f77f32590 WatchSource:0}: Error finding container f3beb9ebd40ca1c38f6f897f6eb6eb878c0b22a9d72592e9a6a3f50f77f32590: Status 404 returned error can't find the container with id f3beb9ebd40ca1c38f6f897f6eb6eb878c0b22a9d72592e9a6a3f50f77f32590
Nov 03 04:02:54.940813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.940776    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg" event={"ID":"870e5933-2258-43e5-900d-2617a3a9650d","Type":"ContainerStarted","Data":"245d204e0cc3c871e34a0d9282d7af4d0655b926616696cfb608af667d95ef53"}
Nov 03 04:02:54.940983 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.940837    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg" event={"ID":"870e5933-2258-43e5-900d-2617a3a9650d","Type":"ContainerStarted","Data":"f3beb9ebd40ca1c38f6f897f6eb6eb878c0b22a9d72592e9a6a3f50f77f32590"}
Nov 03 04:02:54.940983 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:54.940865    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:02:55.063258 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:55.063165    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg" podStartSLOduration=2.063151564 podStartE2EDuration="2.063151564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:02:55.061069352 +0000 UTC m=+3666.291961770" watchObservedRunningTime="2026-11-03 04:02:55.063151564 +0000 UTC m=+3666.294044006"
Nov 03 04:02:58.295256 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:58.295224    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:58.467480 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:58.467446    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:02:59.031526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:02:59.031481    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:04.125184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:04.125153    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:05.950792 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:05.950764    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:03:06.166101 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:06.166064    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:03:09.017421 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:09.016860    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:09.161297 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:09.161261    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:13.810364 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:13.810326    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:18.726514 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:18.726486    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:20.984140 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:03:20.984100    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:03:22.360121 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:22.360090    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:23.614594 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:23.614558    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:23.835534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:23.835502    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:24.043044 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:24.042903    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:27.823128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:27.823090    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:27.997411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:27.997379    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:28.143780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:28.143744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:28.717403 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:28.717371    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:28.860709 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:28.860681    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:31.932914 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:31.932878    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:32.117590 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.117502    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:32.374024 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.373989    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"]
Nov 03 04:03:32.378401 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.378382    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.432979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.432948    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-90139c\""
Nov 03 04:03:32.485528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.485491    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-90139c-dockercfg-mrxsk\""
Nov 03 04:03:32.521576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.519475    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"]
Nov 03 04:03:32.529678 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.529645    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"]
Nov 03 04:03:32.540075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.540038    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.578214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.578186    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"]
Nov 03 04:03:32.586278 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.586240    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7cq\" (UniqueName: \"kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq\") pod \"payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.586472 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.586379    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.687868 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.687779    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7cq\" (UniqueName: \"kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq\") pod \"payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.687868 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.687827    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8ss\" (UniqueName: \"kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss\") pod \"payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.688140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.687878    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.688140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.687906    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.688687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.688664    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.750069 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.750035    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7cq\" (UniqueName: \"kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq\") pod \"payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:32.789184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.789138    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8ss\" (UniqueName: \"kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss\") pod \"payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.789396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.789236    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.789988 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.789968    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.851659 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.851627    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8ss\" (UniqueName: \"kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss\") pod \"payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.859380 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.859350    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:32.992089 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:32.992000    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:33.237520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:33.237489    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"]
Nov 03 04:03:33.238861 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:03:33.238833    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21bc5e4_ba98_4c03_97e1_c81cf4c56d6f.slice/crio-36eff15f07e354d3338fe9d1b95a951bb86402a829258a4cc06eaa042259ac59 WatchSource:0}: Error finding container 36eff15f07e354d3338fe9d1b95a951bb86402a829258a4cc06eaa042259ac59: Status 404 returned error can't find the container with id 36eff15f07e354d3338fe9d1b95a951bb86402a829258a4cc06eaa042259ac59
Nov 03 04:03:33.373762 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:33.373735    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"]
Nov 03 04:03:33.374699 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:03:33.374671    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d2e5e2_85df_4095_b8c6_dde7b133578f.slice/crio-47c4552ea8d46ced9534b7adf37befe78b1e7f23fe675e1f2c96b771d8aeea4c WatchSource:0}: Error finding container 47c4552ea8d46ced9534b7adf37befe78b1e7f23fe675e1f2c96b771d8aeea4c: Status 404 returned error can't find the container with id 47c4552ea8d46ced9534b7adf37befe78b1e7f23fe675e1f2c96b771d8aeea4c
Nov 03 04:03:33.809027 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:33.808999    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:34.024244 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.024200    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:34.165433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.165404    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:34.188994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.188955    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh" event={"ID":"88d2e5e2-85df-4095-b8c6-dde7b133578f","Type":"ContainerStarted","Data":"3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b"}
Nov 03 04:03:34.189184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.189019    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh" event={"ID":"88d2e5e2-85df-4095-b8c6-dde7b133578f","Type":"ContainerStarted","Data":"47c4552ea8d46ced9534b7adf37befe78b1e7f23fe675e1f2c96b771d8aeea4c"}
Nov 03 04:03:34.189184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.189053    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:34.190669 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.190646    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll" event={"ID":"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f","Type":"ContainerStarted","Data":"e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089"}
Nov 03 04:03:34.190794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.190686    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll" event={"ID":"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f","Type":"ContainerStarted","Data":"36eff15f07e354d3338fe9d1b95a951bb86402a829258a4cc06eaa042259ac59"}
Nov 03 04:03:34.190794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.190705    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:34.306374 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.306320    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh" podStartSLOduration=2.306302931 podStartE2EDuration="2.306302931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:34.30611036 +0000 UTC m=+3705.537002800" watchObservedRunningTime="2026-11-03 04:03:34.306302931 +0000 UTC m=+3705.537195350"
Nov 03 04:03:34.442113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:34.442000    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll" podStartSLOduration=2.441980478 podStartE2EDuration="2.441980478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:34.439003319 +0000 UTC m=+3705.669895737" watchObservedRunningTime="2026-11-03 04:03:34.441980478 +0000 UTC m=+3705.672872897"
Nov 03 04:03:37.880450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:37.880418    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:38.815911 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:38.815874    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:39.221815 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:39.221778    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:39.560178 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:39.560091    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:39.809478 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:39.809435    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:43.863411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:43.863381    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:44.047167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:44.047136    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:44.188916 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:44.188843    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:44.586589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:44.586496    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:45.199340 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:45.199314    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:03:45.199790 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:45.199380    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:03:48.432283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.432240    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:48.630596 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.630501    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:48.773424 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.773320    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:48.854409 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.854376    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"]
Nov 03 04:03:48.858658 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.858639    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:48.908928 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.908888    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5h8\" (UniqueName: \"kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8\") pod \"payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:48.909065 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.909045    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:48.916813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.916779    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"]
Nov 03 04:03:48.965511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.965479    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-55dc0c-dockercfg-fwsbb\""
Nov 03 04:03:48.965888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:48.965867    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-55dc0c\""
Nov 03 04:03:49.010891 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.010834    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5h8\" (UniqueName: \"kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8\") pod \"payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:49.011199 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.011167    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:49.012078 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.012048    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:49.073127 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.073058    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5h8\" (UniqueName: \"kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8\") pod \"payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:49.225943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.225909    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-55dc0c-dockercfg-fwsbb\""
Nov 03 04:03:49.234019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.233998    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:49.614699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.614668    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"]
Nov 03 04:03:49.616580 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:03:49.616527    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d25796_a87c_4d09_926f_fc1e9848e93d.slice/crio-6355b1eb68dcc5be9c759c36d0174903916f25ee6a934eb1cfd002733aed39af WatchSource:0}: Error finding container 6355b1eb68dcc5be9c759c36d0174903916f25ee6a934eb1cfd002733aed39af: Status 404 returned error can't find the container with id 6355b1eb68dcc5be9c759c36d0174903916f25ee6a934eb1cfd002733aed39af
Nov 03 04:03:49.849945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:49.849897    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:50.302580 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:50.300654    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk" event={"ID":"01d25796-a87c-4d09-926f-fc1e9848e93d","Type":"ContainerStarted","Data":"02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4"}
Nov 03 04:03:50.302580 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:50.300723    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk" event={"ID":"01d25796-a87c-4d09-926f-fc1e9848e93d","Type":"ContainerStarted","Data":"6355b1eb68dcc5be9c759c36d0174903916f25ee6a934eb1cfd002733aed39af"}
Nov 03 04:03:50.305860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:50.305829    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:03:50.426517 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:50.426461    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk" podStartSLOduration=2.426444895 podStartE2EDuration="2.426444895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:03:50.424465725 +0000 UTC m=+3721.655358139" watchObservedRunningTime="2026-11-03 04:03:50.426444895 +0000 UTC m=+3721.657337313"
Nov 03 04:03:50.986908 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:03:50.986872    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:03:54.649780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:54.649745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:03:59.495747 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:03:59.495711    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:02.316774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:02.316737    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:04:04.352660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:04.352624    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:09.077640 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:09.077603    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:09.242389 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:09.242358    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:09.384409 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:09.384377    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:14.136331 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:14.136298    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:18.985794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:18.985759    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:19.164918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:19.164859    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:19.307739 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:19.307657    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:20.985996 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:04:20.985958    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:04:24.014788 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:24.014754    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:26.545133 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:26.545098    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:04:28.868764 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:28.868731    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:33.733906 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:33.733873    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:37.300700 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:37.300668    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:37.440220 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:37.440186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:37.805138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:37.805094    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:37.944035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:37.943997    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:38.635044 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:38.635003    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:43.536297 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:43.536268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:48.365761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:48.365723    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:50.984442 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:04:50.984402    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:04:53.229903 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:53.229821    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:57.995397 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:57.995361    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:58.181870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:58.181828    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:04:58.323520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:04:58.323434    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:01.281652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:01.281616    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:05:03.142044 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:03.141984    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:08.019128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:08.019092    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:09.667312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:09.667283    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:12.841354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:12.841321    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:16.883330 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:16.883294    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:17.875623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:17.875585    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"]
Nov 03 04:05:17.879717 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:17.879698    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:17.935337 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:17.935301    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-55dc0c-serving-cert\""
Nov 03 04:05:17.988735 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:17.988702    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"]
Nov 03 04:05:18.021101 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.021065    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgwz\" (UniqueName: \"kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.021285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.021128    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.021362 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.021277    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.021476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.021458    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.074077 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.074042    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:18.127515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.127418    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.131608 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.131529    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgwz\" (UniqueName: \"kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.132175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.132151    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.141444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.141415    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.150572 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:05:18.147865    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-serving-cert: secret "maas-api-e2e-autores-55dc0c-serving-cert" not found
Nov 03 04:05:18.150572 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:05:18.147956    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls podName:ebc4695c-2a9b-47c1-9be2-69b8611fca1e nodeName:}" failed. No retries permitted until 2026-11-03 04:05:18.647932203 +0000 UTC m=+3809.878824602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls") pod "maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" (UID: "ebc4695c-2a9b-47c1-9be2-69b8611fca1e") : secret "maas-api-e2e-autores-55dc0c-serving-cert" not found
Nov 03 04:05:18.150572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.141025    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.152614 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.152585    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.204046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.204009    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgwz\" (UniqueName: \"kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.238260 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.238226    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:18.386003 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.385893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:18.653948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.653849    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.656478 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.656457    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\") pod \"maas-api-e2e-autores-55dc0c-67b48988bc-msvqm\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:18.793312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:18.793277    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:19.175113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:19.175072    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"]
Nov 03 04:05:19.176076 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:05:19.176051    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc4695c_2a9b_47c1_9be2_69b8611fca1e.slice/crio-7c90990086658eaa5ba29fa9c21f4b0145ef95eaeac4dd0f5508bc6e2b3ce354 WatchSource:0}: Error finding container 7c90990086658eaa5ba29fa9c21f4b0145ef95eaeac4dd0f5508bc6e2b3ce354: Status 404 returned error can't find the container with id 7c90990086658eaa5ba29fa9c21f4b0145ef95eaeac4dd0f5508bc6e2b3ce354
Nov 03 04:05:19.924143 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:19.924107    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" event={"ID":"ebc4695c-2a9b-47c1-9be2-69b8611fca1e","Type":"ContainerStarted","Data":"7c90990086658eaa5ba29fa9c21f4b0145ef95eaeac4dd0f5508bc6e2b3ce354"}
Nov 03 04:05:20.931915 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:20.931869    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" event={"ID":"ebc4695c-2a9b-47c1-9be2-69b8611fca1e","Type":"ContainerStarted","Data":"165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04"}
Nov 03 04:05:20.932358 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:20.931984    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:20.985021 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:05:20.984982    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:05:21.052222 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:21.052170    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" podStartSLOduration=3.283362243 podStartE2EDuration="4.052153095s" totalImagesPullingTime="768.790852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:05:21.051609016 +0000 UTC m=+3812.282501435" watchObservedRunningTime="2026-11-03 04:05:21.052153095 +0000 UTC m=+3812.283045514"
Nov 03 04:05:23.225916 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:23.225884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:23.450456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:23.450422    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:26.110871 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:26.110833    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:26.316795 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:26.316745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:26.944456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:26.944429    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:05:28.598728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:28.598687    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:29.285428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:29.285395    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:33.718426 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:33.718377    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:38.983542 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:38.982896    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:43.740485 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:43.740442    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:43.980730 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:43.980695    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:05:45.791190 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:45.791158    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:48.598148 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:48.598111    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:50.985733 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:05:50.985691    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:05:51.923212 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:51.923174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:51.976611 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:51.976542    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"]
Nov 03 04:05:52.108592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:52.108543    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:52.452366 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:52.452336    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:52.596309 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:52.596263    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:53.438869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:53.438826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:56.977089 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:56.977052    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg" podUID="870e5933-2258-43e5-900d-2617a3a9650d" containerName="payload-processing" containerID="cri-o://245d204e0cc3c871e34a0d9282d7af4d0655b926616696cfb608af667d95ef53" gracePeriod=25
Nov 03 04:05:57.156431 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.156396    5515 generic.go:356] "Generic (PLEG): container finished" podID="870e5933-2258-43e5-900d-2617a3a9650d" containerID="245d204e0cc3c871e34a0d9282d7af4d0655b926616696cfb608af667d95ef53" exitCode=0
Nov 03 04:05:57.156666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.156643    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg" event={"ID":"870e5933-2258-43e5-900d-2617a3a9650d","Type":"ContainerDied","Data":"245d204e0cc3c871e34a0d9282d7af4d0655b926616696cfb608af667d95ef53"}
Nov 03 04:05:57.253906 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.253882    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:05:57.346908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.346876    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume\") pod \"870e5933-2258-43e5-900d-2617a3a9650d\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") "
Nov 03 04:05:57.347070 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.346991    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx\" (UniqueName: \"kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx\") pod \"870e5933-2258-43e5-900d-2617a3a9650d\" (UID: \"870e5933-2258-43e5-900d-2617a3a9650d\") "
Nov 03 04:05:57.347240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.347220    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume" pod "870e5933-2258-43e5-900d-2617a3a9650d" (UID: "870e5933-2258-43e5-900d-2617a3a9650d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:05:57.347398 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.347384    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/870e5933-2258-43e5-900d-2617a3a9650d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:05:57.349182 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.349158    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx" pod "870e5933-2258-43e5-900d-2617a3a9650d" (UID: "870e5933-2258-43e5-900d-2617a3a9650d"). InnerVolumeSpecName "kube-api-access-2k9qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:05:57.448171 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.448138    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9qx\" (UniqueName: \"kubernetes.io/projected/870e5933-2258-43e5-900d-2617a3a9650d-kube-api-access-2k9qx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:05:57.642364 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.642330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:57.791270 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:57.791237    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:58.166519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.166490    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg" event={"ID":"870e5933-2258-43e5-900d-2617a3a9650d","Type":"ContainerDied","Data":"f3beb9ebd40ca1c38f6f897f6eb6eb878c0b22a9d72592e9a6a3f50f77f32590"}
Nov 03 04:05:58.166519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.166515    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"
Nov 03 04:05:58.167053 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.166569    5515 scope.go:122] "RemoveContainer" containerID="245d204e0cc3c871e34a0d9282d7af4d0655b926616696cfb608af667d95ef53"
Nov 03 04:05:58.333431 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.333396    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"]
Nov 03 04:05:58.384141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.384112    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c832d-5f8d7dc8c9-tzbcg"]
Nov 03 04:05:58.505986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.505878    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:58.654581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.654522    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:05:58.895364 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:58.895328    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870e5933-2258-43e5-900d-2617a3a9650d" path="/var/lib/kubelet/pods/870e5933-2258-43e5-900d-2617a3a9650d/volumes"
Nov 03 04:05:59.417577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:05:59.417520    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:04.252498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:04.252466    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:04.402513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:04.402480    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:05.995715 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:05.995678    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:06.663088 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:06.663057    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:09.637006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:09.636973    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:10.027329 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:10.027244    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:10.701373 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:10.701341    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:10.853962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:10.853619    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:14.632637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:14.632599    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:14.790177 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:14.790148    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:19.544870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:19.544833    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:20.984867 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:06:20.984816    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:06:23.667162 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:23.667082    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"]
Nov 03 04:06:23.667723 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:23.667328    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" podUID="3a08833e-102e-48b1-a16a-54de75ea98e9" containerName="maas-api" containerID="cri-o://82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447" gracePeriod=30
Nov 03 04:06:23.769900 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:23.769866    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:23.991011 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:23.990987    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:06:24.017425 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.017394    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg\" (UniqueName: \"kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg\") pod \"3a08833e-102e-48b1-a16a-54de75ea98e9\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") "
Nov 03 04:06:24.017597 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.017453    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs\") pod \"3a08833e-102e-48b1-a16a-54de75ea98e9\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") "
Nov 03 04:06:24.017597 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.017572    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca\") pod \"3a08833e-102e-48b1-a16a-54de75ea98e9\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") "
Nov 03 04:06:24.017751 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.017655    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\") pod \"3a08833e-102e-48b1-a16a-54de75ea98e9\" (UID: \"3a08833e-102e-48b1-a16a-54de75ea98e9\") "
Nov 03 04:06:24.017994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.017958    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca" pod "3a08833e-102e-48b1-a16a-54de75ea98e9" (UID: "3a08833e-102e-48b1-a16a-54de75ea98e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:06:24.018214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.018186    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a08833e-102e-48b1-a16a-54de75ea98e9-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:24.019787 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.019766    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg" pod "3a08833e-102e-48b1-a16a-54de75ea98e9" (UID: "3a08833e-102e-48b1-a16a-54de75ea98e9"). InnerVolumeSpecName "kube-api-access-2djgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:06:24.019862 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.019784    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs" pod "3a08833e-102e-48b1-a16a-54de75ea98e9" (UID: "3a08833e-102e-48b1-a16a-54de75ea98e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:06:24.019917 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.019857    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls" pod "3a08833e-102e-48b1-a16a-54de75ea98e9" (UID: "3a08833e-102e-48b1-a16a-54de75ea98e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:06:24.119515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.119476    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2djgg\" (UniqueName: \"kubernetes.io/projected/3a08833e-102e-48b1-a16a-54de75ea98e9-kube-api-access-2djgg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:24.119515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.119510    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:24.119515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.119520    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a08833e-102e-48b1-a16a-54de75ea98e9-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:24.333858 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.333764    5515 generic.go:356] "Generic (PLEG): container finished" podID="3a08833e-102e-48b1-a16a-54de75ea98e9" containerID="82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447" exitCode=0
Nov 03 04:06:24.333858 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.333809    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"
Nov 03 04:06:24.334106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.333854    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" event={"ID":"3a08833e-102e-48b1-a16a-54de75ea98e9","Type":"ContainerDied","Data":"82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447"}
Nov 03 04:06:24.334106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.333909    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4" event={"ID":"3a08833e-102e-48b1-a16a-54de75ea98e9","Type":"ContainerDied","Data":"87c917188d9c7cf1582d46387b1badb4bf86a84cb78d4c2e696bd89c402897a4"}
Nov 03 04:06:24.334106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.333939    5515 scope.go:122] "RemoveContainer" containerID="82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447"
Nov 03 04:06:24.350309 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.350280    5515 scope.go:122] "RemoveContainer" containerID="82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447"
Nov 03 04:06:24.350706 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:06:24.350678    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447\": container with ID starting with 82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447 not found: ID does not exist" containerID="82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447"
Nov 03 04:06:24.350828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.350717    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447"} err="failed to get container status \"82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447\": rpc error: code = NotFound desc = could not find container \"82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447\": container with ID starting with 82c79e389bb7da97c9d3077fafead11dcf92ce04ea7f8b42fb448d6119af8447 not found: ID does not exist"
Nov 03 04:06:24.524358 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.524324    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"]
Nov 03 04:06:24.526702 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.526681    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5172b1-c497db4b6-42cr4"]
Nov 03 04:06:24.895672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:24.895642    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a08833e-102e-48b1-a16a-54de75ea98e9" path="/var/lib/kubelet/pods/3a08833e-102e-48b1-a16a-54de75ea98e9/volumes"
Nov 03 04:06:25.623330 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:25.623299    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:27.210878 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:27.210850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:27.335426 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:27.335377    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:06:29.435053 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:29.435007    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:29.581884 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:29.581849    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:30.563891 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:30.563819    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:31.089175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:31.089118    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:31.596205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:31.596171    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:32.418558 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:32.418525    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:32.758073 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:32.758000    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"]
Nov 03 04:06:32.871661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:32.871632    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"]
Nov 03 04:06:33.121833 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:33.121802    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:33.266143 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:33.266112    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:34.221182 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:34.221143    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:35.831847 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:35.831815    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:37.758465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:37.758428    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll" podUID="c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" containerName="payload-processing" containerID="cri-o://e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089" gracePeriod=25
Nov 03 04:06:37.872782 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:37.872738    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh" podUID="88d2e5e2-85df-4095-b8c6-dde7b133578f" containerName="payload-pre-processing" containerID="cri-o://3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b" gracePeriod=25
Nov 03 04:06:38.044149 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.044124    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:06:38.129239 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.129051    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss\" (UniqueName: \"kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss\") pod \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") "
Nov 03 04:06:38.129239 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.129175    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume\") pod \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\" (UID: \"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f\") "
Nov 03 04:06:38.129598 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.129573    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume" pod "c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" (UID: "c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:06:38.129696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.129682    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:38.131315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.131290    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss" pod "c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" (UID: "c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f"). InnerVolumeSpecName "kube-api-access-bc8ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:06:38.141863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.141838    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:06:38.207943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.207912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:38.230686 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.230661    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume\") pod \"88d2e5e2-85df-4095-b8c6-dde7b133578f\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") "
Nov 03 04:06:38.230795 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.230737    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq\" (UniqueName: \"kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq\") pod \"88d2e5e2-85df-4095-b8c6-dde7b133578f\" (UID: \"88d2e5e2-85df-4095-b8c6-dde7b133578f\") "
Nov 03 04:06:38.231038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.231018    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume" pod "88d2e5e2-85df-4095-b8c6-dde7b133578f" (UID: "88d2e5e2-85df-4095-b8c6-dde7b133578f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:06:38.231163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.231150    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d2e5e2-85df-4095-b8c6-dde7b133578f-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:38.231209 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.231166    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8ss\" (UniqueName: \"kubernetes.io/projected/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f-kube-api-access-bc8ss\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:38.232871 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.232853    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq" pod "88d2e5e2-85df-4095-b8c6-dde7b133578f" (UID: "88d2e5e2-85df-4095-b8c6-dde7b133578f"). InnerVolumeSpecName "kube-api-access-zh7cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:06:38.332925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.332838    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7cq\" (UniqueName: \"kubernetes.io/projected/88d2e5e2-85df-4095-b8c6-dde7b133578f-kube-api-access-zh7cq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:38.353146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.353121    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:38.432004 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.431965    5515 generic.go:356] "Generic (PLEG): container finished" podID="c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" containerID="e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089" exitCode=0
Nov 03 04:06:38.432218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.432078    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"
Nov 03 04:06:38.432218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.432099    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll" event={"ID":"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f","Type":"ContainerDied","Data":"e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089"}
Nov 03 04:06:38.432218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.432166    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll" event={"ID":"c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f","Type":"ContainerDied","Data":"36eff15f07e354d3338fe9d1b95a951bb86402a829258a4cc06eaa042259ac59"}
Nov 03 04:06:38.432218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.432200    5515 scope.go:122] "RemoveContainer" containerID="e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089"
Nov 03 04:06:38.433605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.433586    5515 generic.go:356] "Generic (PLEG): container finished" podID="88d2e5e2-85df-4095-b8c6-dde7b133578f" containerID="3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b" exitCode=0
Nov 03 04:06:38.433770 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.433753    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"
Nov 03 04:06:38.433869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.433738    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh" event={"ID":"88d2e5e2-85df-4095-b8c6-dde7b133578f","Type":"ContainerDied","Data":"3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b"}
Nov 03 04:06:38.433939 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.433927    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh" event={"ID":"88d2e5e2-85df-4095-b8c6-dde7b133578f","Type":"ContainerDied","Data":"47c4552ea8d46ced9534b7adf37befe78b1e7f23fe675e1f2c96b771d8aeea4c"}
Nov 03 04:06:38.444252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.444233    5515 scope.go:122] "RemoveContainer" containerID="e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089"
Nov 03 04:06:38.444533 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:06:38.444516    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089\": container with ID starting with e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089 not found: ID does not exist" containerID="e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089"
Nov 03 04:06:38.444616 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.444541    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089"} err="failed to get container status \"e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089\": rpc error: code = NotFound desc = could not find container \"e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089\": container with ID starting with e7046df0114c505f08c2dff123ef02d11dd8b9a779777c8619269000397a3089 not found: ID does not exist"
Nov 03 04:06:38.444616 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.444580    5515 scope.go:122] "RemoveContainer" containerID="3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b"
Nov 03 04:06:38.454232 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.454216    5515 scope.go:122] "RemoveContainer" containerID="3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b"
Nov 03 04:06:38.454505 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:06:38.454484    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b\": container with ID starting with 3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b not found: ID does not exist" containerID="3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b"
Nov 03 04:06:38.454597 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.454513    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b"} err="failed to get container status \"3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b\": rpc error: code = NotFound desc = could not find container \"3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b\": container with ID starting with 3f6d61ad5312204b206bb4fc44484a81dc1ecd8a71f0108fd9890a621509ff9b not found: ID does not exist"
Nov 03 04:06:38.601911 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.601836    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"]
Nov 03 04:06:38.609097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.609073    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-90139c-665bf56fd7-wrdgh"]
Nov 03 04:06:38.722199 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.722172    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"]
Nov 03 04:06:38.729807 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.729785    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-90139c-557659cd9-6qrll"]
Nov 03 04:06:38.892864 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.892799    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:38.896752 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.896730    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d2e5e2-85df-4095-b8c6-dde7b133578f" path="/var/lib/kubelet/pods/88d2e5e2-85df-4095-b8c6-dde7b133578f/volumes"
Nov 03 04:06:38.897125 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:38.897113    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" path="/var/lib/kubelet/pods/c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f/volumes"
Nov 03 04:06:40.215619 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:40.215584    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:40.359685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:40.359658    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:42.232912 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:42.232881    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:43.033474 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:43.033440    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:43.180141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:43.180107    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:43.667884 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:43.667851    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:44.907755 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:44.907721    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:47.589620 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:47.589579    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:49.962137 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:49.962092    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:50.251144 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:50.251076    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:06:50.253255 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:50.253233    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:06:50.287286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:50.287264    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:06:50.289774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:50.289752    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:06:50.756758 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:50.756726    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"]
Nov 03 04:06:50.985721 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:06:50.985682    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:06:51.011013 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:51.010933    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:51.164275 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:51.164244    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:52.462942 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:52.462897    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:55.757479 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:55.757403    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk" podUID="01d25796-a87c-4d09-926f-fc1e9848e93d" containerName="payload-pre-processing" containerID="cri-o://02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4" gracePeriod=25
Nov 03 04:06:56.031306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.031276    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:06:56.099254 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.099224    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume\") pod \"01d25796-a87c-4d09-926f-fc1e9848e93d\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") "
Nov 03 04:06:56.099444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.099270    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8\" (UniqueName: \"kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8\") pod \"01d25796-a87c-4d09-926f-fc1e9848e93d\" (UID: \"01d25796-a87c-4d09-926f-fc1e9848e93d\") "
Nov 03 04:06:56.099634 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.099605    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume" pod "01d25796-a87c-4d09-926f-fc1e9848e93d" (UID: "01d25796-a87c-4d09-926f-fc1e9848e93d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:06:56.099744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.099728    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d25796-a87c-4d09-926f-fc1e9848e93d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:56.101981 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.101950    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8" pod "01d25796-a87c-4d09-926f-fc1e9848e93d" (UID: "01d25796-a87c-4d09-926f-fc1e9848e93d"). InnerVolumeSpecName "kube-api-access-ch5h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:06:56.200632 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.200602    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5h8\" (UniqueName: \"kubernetes.io/projected/01d25796-a87c-4d09-926f-fc1e9848e93d-kube-api-access-ch5h8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:06:56.508576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.508505    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:06:56.546265 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.546235    5515 generic.go:356] "Generic (PLEG): container finished" podID="01d25796-a87c-4d09-926f-fc1e9848e93d" containerID="02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4" exitCode=0
Nov 03 04:06:56.546451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.546279    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"
Nov 03 04:06:56.546451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.546324    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk" event={"ID":"01d25796-a87c-4d09-926f-fc1e9848e93d","Type":"ContainerDied","Data":"02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4"}
Nov 03 04:06:56.546451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.546382    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk" event={"ID":"01d25796-a87c-4d09-926f-fc1e9848e93d","Type":"ContainerDied","Data":"6355b1eb68dcc5be9c759c36d0174903916f25ee6a934eb1cfd002733aed39af"}
Nov 03 04:06:56.546451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.546411    5515 scope.go:122] "RemoveContainer" containerID="02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4"
Nov 03 04:06:56.556320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.556298    5515 scope.go:122] "RemoveContainer" containerID="02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4"
Nov 03 04:06:56.556610 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:06:56.556590    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4\": container with ID starting with 02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4 not found: ID does not exist" containerID="02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4"
Nov 03 04:06:56.556709 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.556615    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4"} err="failed to get container status \"02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4\": rpc error: code = NotFound desc = could not find container \"02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4\": container with ID starting with 02955ee21158449290f0f1e71ac138ad116786282e889e6bd3f3387bf56cd5d4 not found: ID does not exist"
Nov 03 04:06:56.715122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.715083    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"]
Nov 03 04:06:56.766046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.765985    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-55dc0c-7948fb749f-jdjjk"]
Nov 03 04:06:56.894910 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:56.894883    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d25796-a87c-4d09-926f-fc1e9848e93d" path="/var/lib/kubelet/pods/01d25796-a87c-4d09-926f-fc1e9848e93d/volumes"
Nov 03 04:06:57.320851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:06:57.320819    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:02.159694 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:02.159658    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:07.141587 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:07.141539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:10.378575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:10.378509    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:12.051521 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:12.051486    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:17.039356 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:17.039323    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:20.984563 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:07:20.984509    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:07:22.029973 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:22.029939    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:26.920507 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:26.920467    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:31.817055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:31.817019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:34.677293 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:34.677258    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:07:36.666564 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:36.666525    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:41.526827 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:41.526795    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:46.409252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:46.409192    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:48.114800 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.114770    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"]
Nov 03 04:07:48.115322 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115308    5515 cpu_manager.go:414] "containerMap: removing container" podUID="870e5933-2258-43e5-900d-2617a3a9650d" containerName="payload-processing"
Nov 03 04:07:48.115368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115323    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870e5933-2258-43e5-900d-2617a3a9650d" containerName="payload-processing"
Nov 03 04:07:48.115368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115346    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" containerName="payload-processing"
Nov 03 04:07:48.115368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115351    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" containerName="payload-processing"
Nov 03 04:07:48.115368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115362    5515 cpu_manager.go:414] "containerMap: removing container" podUID="88d2e5e2-85df-4095-b8c6-dde7b133578f" containerName="payload-pre-processing"
Nov 03 04:07:48.115368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115368    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d2e5e2-85df-4095-b8c6-dde7b133578f" containerName="payload-pre-processing"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115390    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a08833e-102e-48b1-a16a-54de75ea98e9" containerName="maas-api"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115395    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a08833e-102e-48b1-a16a-54de75ea98e9" containerName="maas-api"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115412    5515 cpu_manager.go:414] "containerMap: removing container" podUID="01d25796-a87c-4d09-926f-fc1e9848e93d" containerName="payload-pre-processing"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115417    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d25796-a87c-4d09-926f-fc1e9848e93d" containerName="payload-pre-processing"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115497    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="870e5933-2258-43e5-900d-2617a3a9650d" containerName="payload-processing"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115507    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21bc5e4-ba98-4c03-97e1-c81cf4c56d6f" containerName="payload-processing"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115517    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d2e5e2-85df-4095-b8c6-dde7b133578f" containerName="payload-pre-processing"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115527    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a08833e-102e-48b1-a16a-54de75ea98e9" containerName="maas-api"
Nov 03 04:07:48.115537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.115535    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d25796-a87c-4d09-926f-fc1e9848e93d" containerName="payload-pre-processing"
Nov 03 04:07:48.118855 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.118832    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.165411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.165382    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"]
Nov 03 04:07:48.169202 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.169179    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.220756 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.220730    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"]
Nov 03 04:07:48.222744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222718    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.222836 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222753    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.222836 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222772    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5424\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.222836 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222826    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.222997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222844    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.222997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222860    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.222997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.222975    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.223102 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223021    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.223102 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223068    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.223102 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223093    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.223204 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223109    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.223204 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223140    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.223280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223203    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.223280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223223    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.223280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223239    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.223280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223256    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.223417 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223360    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.223417 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.223392    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44lc\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.273371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.273340    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"]
Nov 03 04:07:48.325033 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325001    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325181 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325054    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325181 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325078    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325181 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325103    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325181 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325120    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325181 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325161    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325215    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325246    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325270    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325297    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325349    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325393    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44lc\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325478    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325504    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325506    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325527    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5424\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325532    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325605    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325703    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325757    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.325907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325783    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.326330 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.325920    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.326330 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.326191    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.326330 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.326278    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.326511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.326413    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.326740 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.326714    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.326843 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.326764    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.327047 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.326990    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.327749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.327723    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.327837 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.327762    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.328498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.328474    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.329002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.328986    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.387668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.387603    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.387868 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.387754    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44lc\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc\") pod \"maas-default-gateway-openshift-default-76f4876667-c9ztz\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.387868 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.387852    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.388172 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.388156    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5424\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424\") pod \"maas-default-gateway-openshift-default-76f4876667-rzzkp\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.432985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.432962    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:48.482098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.482070    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:48.865977 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.865945    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"]
Nov 03 04:07:48.867461 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:07:48.867437    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246d8290_3d5f_405a_96bb_60964e3d66c6.slice/crio-7258976b63393d29c833862cadd7167014bf1399c4bee40f6f1ec5496a6bdf7c WatchSource:0}: Error finding container 7258976b63393d29c833862cadd7167014bf1399c4bee40f6f1ec5496a6bdf7c: Status 404 returned error can't find the container with id 7258976b63393d29c833862cadd7167014bf1399c4bee40f6f1ec5496a6bdf7c
Nov 03 04:07:48.869801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.869769    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:07:48.869888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.869830    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:07:48.869888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.869858    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:07:48.918594 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.918572    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"]
Nov 03 04:07:48.919907 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:07:48.919881    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc445bad5_3c95_4631_b207_996f78b0d51e.slice/crio-1654748df0c1c4732129112ea24607ba0e0b87a37c46efb0f3fc0f7e82802a0f WatchSource:0}: Error finding container 1654748df0c1c4732129112ea24607ba0e0b87a37c46efb0f3fc0f7e82802a0f: Status 404 returned error can't find the container with id 1654748df0c1c4732129112ea24607ba0e0b87a37c46efb0f3fc0f7e82802a0f
Nov 03 04:07:48.922985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.922950    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:07:48.923053 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.923038    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:07:48.923099 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:48.923079    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:07:49.893631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:49.890740    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" event={"ID":"c445bad5-3c95-4631-b207-996f78b0d51e","Type":"ContainerStarted","Data":"6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24"}
Nov 03 04:07:49.893631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:49.890804    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" event={"ID":"c445bad5-3c95-4631-b207-996f78b0d51e","Type":"ContainerStarted","Data":"1654748df0c1c4732129112ea24607ba0e0b87a37c46efb0f3fc0f7e82802a0f"}
Nov 03 04:07:49.894219 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:49.893808    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" event={"ID":"246d8290-3d5f-405a-96bb-60964e3d66c6","Type":"ContainerStarted","Data":"1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82"}
Nov 03 04:07:49.894219 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:49.893865    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" event={"ID":"246d8290-3d5f-405a-96bb-60964e3d66c6","Type":"ContainerStarted","Data":"7258976b63393d29c833862cadd7167014bf1399c4bee40f6f1ec5496a6bdf7c"}
Nov 03 04:07:50.014262 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.014206    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podStartSLOduration=2.014187647 podStartE2EDuration="2.014187647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:07:50.013185263 +0000 UTC m=+3961.244077683" watchObservedRunningTime="2026-11-03 04:07:50.014187647 +0000 UTC m=+3961.245080064"
Nov 03 04:07:50.140562 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.140496    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podStartSLOduration=2.140483006 podStartE2EDuration="2.140483006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:07:50.138600767 +0000 UTC m=+3961.369493185" watchObservedRunningTime="2026-11-03 04:07:50.140483006 +0000 UTC m=+3961.371375424"
Nov 03 04:07:50.434038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.433994    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:50.435970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.435946    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:50.436119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.435989    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:50.489576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.488612    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:50.490041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.490011    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:50.490259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.490233    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:50.658307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:50.658257    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:51.142892 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:07:51.142834    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:07:51.434420 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:51.434343    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:51.434420 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:51.434406    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:51.491689 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:51.491652    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:51.491956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:51.491926    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:52.433383 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:52.433287    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:52.433383 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:52.433349    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:52.486577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:52.483038    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:52.486577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:52.483105    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:52.545009 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:52.544979    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:53.433282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:53.433244    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:53.433517 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:53.433320    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:53.487573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:53.483897    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:53.487573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:53.483957    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:54.435573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:54.435168    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:54.435573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:54.435229    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:54.483096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:54.483058    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:54.483410 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:54.483369    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:55.434236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:55.434200    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:55.434444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:55.434263    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:55.483456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:55.483412    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:55.484014 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:55.483480    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:56.434190 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:56.434150    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:07:56.434374 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:56.434215    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:07:56.482807 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:56.482774    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 04:07:56.482967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:56.482832    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 04:07:58.003698 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.003661    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:58.005229 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.005204    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:58.006302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.006282    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:07:58.103583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.103533    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:58.103866 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.103846    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:58.104886 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.104856    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:07:58.383653 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.383620    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:58.523292 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.523263    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:58.841517 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:58.841449    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:59.043214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:59.043186    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:07:59.619182 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:07:59.619142    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:00.062266 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:00.062177    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:00.211602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:00.211530    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:01.036122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:01.036093    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:01.185167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:01.185130    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:02.533445 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:02.533411    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:03.156775 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.156738    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:03.386886 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.386829    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:03.527981 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.527884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:03.595586 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.595533    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"]
Nov 03 04:08:03.596016 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.595777    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" podUID="ebc4695c-2a9b-47c1-9be2-69b8611fca1e" containerName="maas-api" containerID="cri-o://165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04" gracePeriod=30
Nov 03 04:08:03.689707 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.689677    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:03.899583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:03.899560    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:08:04.011677 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.011649    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:04.039896 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.039864    5515 generic.go:356] "Generic (PLEG): container finished" podID="ebc4695c-2a9b-47c1-9be2-69b8611fca1e" containerID="165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04" exitCode=0
Nov 03 04:08:04.040078 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.039946    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"
Nov 03 04:08:04.040078 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.039965    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" event={"ID":"ebc4695c-2a9b-47c1-9be2-69b8611fca1e","Type":"ContainerDied","Data":"165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04"}
Nov 03 04:08:04.040078 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.040018    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm" event={"ID":"ebc4695c-2a9b-47c1-9be2-69b8611fca1e","Type":"ContainerDied","Data":"7c90990086658eaa5ba29fa9c21f4b0145ef95eaeac4dd0f5508bc6e2b3ce354"}
Nov 03 04:08:04.040222 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.040088    5515 scope.go:122] "RemoveContainer" containerID="165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04"
Nov 03 04:08:04.050209 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.050185    5515 scope.go:122] "RemoveContainer" containerID="165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04"
Nov 03 04:08:04.050543 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:08:04.050522    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04\": container with ID starting with 165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04 not found: ID does not exist" containerID="165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04"
Nov 03 04:08:04.050543 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.050581    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04"} err="failed to get container status \"165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04\": rpc error: code = NotFound desc = could not find container \"165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04\": container with ID starting with 165a866c06a7ec68e6cdbea19de177273994b93787d23602b7764701f4ac1d04 not found: ID does not exist"
Nov 03 04:08:04.050772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.050622    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\") pod \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") "
Nov 03 04:08:04.050772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.050730    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca\") pod \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") "
Nov 03 04:08:04.050864 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.050846    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz\" (UniqueName: \"kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz\") pod \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") "
Nov 03 04:08:04.050921 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.050900    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs\") pod \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\" (UID: \"ebc4695c-2a9b-47c1-9be2-69b8611fca1e\") "
Nov 03 04:08:04.051111 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.051090    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca" pod "ebc4695c-2a9b-47c1-9be2-69b8611fca1e" (UID: "ebc4695c-2a9b-47c1-9be2-69b8611fca1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:08:04.051640 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.051585    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:08:04.053320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.053287    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs" pod "ebc4695c-2a9b-47c1-9be2-69b8611fca1e" (UID: "ebc4695c-2a9b-47c1-9be2-69b8611fca1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:08:04.053591 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.053537    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz" pod "ebc4695c-2a9b-47c1-9be2-69b8611fca1e" (UID: "ebc4695c-2a9b-47c1-9be2-69b8611fca1e"). InnerVolumeSpecName "kube-api-access-jcgwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:08:04.053976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.053946    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls" pod "ebc4695c-2a9b-47c1-9be2-69b8611fca1e" (UID: "ebc4695c-2a9b-47c1-9be2-69b8611fca1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:08:04.156187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.155666    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:08:04.156187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.155704    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:08:04.156187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.155718    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgwz\" (UniqueName: \"kubernetes.io/projected/ebc4695c-2a9b-47c1-9be2-69b8611fca1e-kube-api-access-jcgwz\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:08:04.466283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.466199    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"]
Nov 03 04:08:04.474757 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.474729    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55dc0c-67b48988bc-msvqm"]
Nov 03 04:08:04.518354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.518320    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:04.871312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.871286    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:04.895193 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:04.895160    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc4695c-2a9b-47c1-9be2-69b8611fca1e" path="/var/lib/kubelet/pods/ebc4695c-2a9b-47c1-9be2-69b8611fca1e/volumes"
Nov 03 04:08:05.500794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:05.500744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:05.655579 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:05.649041    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:06.242248 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:06.242211    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:06.495620 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:06.492162    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:07.100417 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:07.100390    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:07.315814 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:07.315753    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:07.460086 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:07.460053    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:07.599377 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:07.599344    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:08.825960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:08.825928    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:09.077656 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:09.077576    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:09.221165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:09.221130    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:09.566220 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:09.566163    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:10.617411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:10.617375    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:10.766340 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:10.766295    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:10.920433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:10.920358    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:11.073423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:11.073388    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:11.675503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:11.675464    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:11.836609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:11.836567    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:11.980252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:11.980166    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:13.637696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:13.637630    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:13.779708 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:13.779679    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:15.455317 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:15.455283    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:15.641600 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:15.638391    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:16.427991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:16.427959    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:16.585782 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:16.585726    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:16.826369 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:16.826247    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:18.012507 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:18.012474    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:20.225586 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:20.225544    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:20.374731 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:20.374688    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:20.532727 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:20.523541    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:20.985456 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:08:20.985421    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:08:21.521051 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:21.521022    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:21.662008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:21.661975    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:22.549103 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:22.549072    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:22.691933 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:22.691900    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:23.025582 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:23.019841    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:23.229151 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:23.216888    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:26.818247 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:26.818215    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:27.974665 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:27.974629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:28.149046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.149019    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:28.402099 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.402065    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"]
Nov 03 04:08:28.402710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.402696    5515 cpu_manager.go:414] "containerMap: removing container" podUID="ebc4695c-2a9b-47c1-9be2-69b8611fca1e" containerName="maas-api"
Nov 03 04:08:28.402769 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.402712    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc4695c-2a9b-47c1-9be2-69b8611fca1e" containerName="maas-api"
Nov 03 04:08:28.402825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.402816    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc4695c-2a9b-47c1-9be2-69b8611fca1e" containerName="maas-api"
Nov 03 04:08:28.406400 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.406382    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.460249 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.460223    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c4fa79\""
Nov 03 04:08:28.460398 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.460304    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c4fa79-dockercfg-tl7nb\""
Nov 03 04:08:28.464781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.464751    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"]
Nov 03 04:08:28.592574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.591948    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czcth\" (UniqueName: \"kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth\") pod \"payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.592574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.592104    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.609647 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.600967    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"]
Nov 03 04:08:28.609647 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.606781    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.622575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.618941    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"]
Nov 03 04:08:28.693585 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.693496    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6s2v\" (UniqueName: \"kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v\") pod \"payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.693770 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.693587    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czcth\" (UniqueName: \"kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth\") pod \"payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.693770 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.693702    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.693770 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.693729    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.694381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.694361    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.755617 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.755589    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czcth\" (UniqueName: \"kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth\") pod \"payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:28.795323 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.795297    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6s2v\" (UniqueName: \"kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v\") pod \"payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.795471 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.795403    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.796043 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.796024    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.857214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.857185    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6s2v\" (UniqueName: \"kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v\") pod \"payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:28.936812 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:28.936784    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:29.020303 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.020100    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:29.178442 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.178400    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:29.314264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.314235    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"]
Nov 03 04:08:29.314470 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:08:29.314447    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de58841_d1a2_4027_b005_65cb7506c1e3.slice/crio-39201316644b64552d117607334c7b68428424068e101fe51bd321c1843a29f5 WatchSource:0}: Error finding container 39201316644b64552d117607334c7b68428424068e101fe51bd321c1843a29f5: Status 404 returned error can't find the container with id 39201316644b64552d117607334c7b68428424068e101fe51bd321c1843a29f5
Nov 03 04:08:29.320312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.320281    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:29.323265 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.323243    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr" event={"ID":"0de58841-d1a2-4027-b005-65cb7506c1e3","Type":"ContainerStarted","Data":"39201316644b64552d117607334c7b68428424068e101fe51bd321c1843a29f5"}
Nov 03 04:08:29.461952 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.453841    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"]
Nov 03 04:08:29.481588 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:29.467680    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:30.334470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.334433    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr" event={"ID":"0de58841-d1a2-4027-b005-65cb7506c1e3","Type":"ContainerStarted","Data":"9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5"}
Nov 03 04:08:30.334947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.334625    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:30.336079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.336058    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk" event={"ID":"9fd73f79-7c61-4766-b7bd-4e8393d612c5","Type":"ContainerStarted","Data":"d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832"}
Nov 03 04:08:30.336165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.336095    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk" event={"ID":"9fd73f79-7c61-4766-b7bd-4e8393d612c5","Type":"ContainerStarted","Data":"56e2743e3451d23a5a73d7f032555975f4ebde70ce890fd338f100a9a1ecea95"}
Nov 03 04:08:30.336165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.336116    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:30.455367 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.455312    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr" podStartSLOduration=2.455299882 podStartE2EDuration="2.455299882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:30.455063386 +0000 UTC m=+4001.685955804" watchObservedRunningTime="2026-11-03 04:08:30.455299882 +0000 UTC m=+4001.686192300"
Nov 03 04:08:30.594606 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:30.590441    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk" podStartSLOduration=2.590416717 podStartE2EDuration="2.590416717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:30.582853323 +0000 UTC m=+4001.813745756" watchObservedRunningTime="2026-11-03 04:08:30.590416717 +0000 UTC m=+4001.821309137"
Nov 03 04:08:31.075280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:31.075250    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:32.608196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:32.608160    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:32.782616 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:32.782585    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:32.930875 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:32.930800    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:33.169513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.169479    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"]
Nov 03 04:08:33.173826 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.173805    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.310572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.309893    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-934bd4-openshift-default-dockercfg-hwwgt\""
Nov 03 04:08:33.329243 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.329206    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"]
Nov 03 04:08:33.334759 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.332241    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335016    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335181    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335310 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335297    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335414 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335403    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335501 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335488    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqhv\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335680    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335805 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335793    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.335913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.335901    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.386869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.386829    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:33.436876 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.436806    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.436876 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.436841    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqhv\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437059 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.436896    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437059 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.436913    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437059 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.436939    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437059 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437046    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437235 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437098    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437235 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437140    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437235 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437184    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437367    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437475    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437488    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437510    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.437872 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.437852    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.439830 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.439791    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.439949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.439840    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.546412 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.546377    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.557429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.554542    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"]
Nov 03 04:08:33.560086 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.560051    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:33.594775 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.594731    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqhv\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv\") pod \"e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") " pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.626440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.620381    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"]
Nov 03 04:08:33.626440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.620465    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.645632 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.645250    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.645632 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.645307    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.645632 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.645397    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.646285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.646097    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.646285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.646141    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.646285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.646209    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.649604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.649288    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmbm\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.649604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.649362    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.649604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.649404    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750377 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750304    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750377 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750347    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750402    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750420    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750471    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750592    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750623    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750682    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750695    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750722    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmbm\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750751    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.750970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.750821    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.751161 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.751015    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.751461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.751443    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.752929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.752910    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.753173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.753153    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.788828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.788802    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:33.812651 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.812623    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmbm\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.812810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.812788    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-rbh9t\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:33.958009 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:33.957976    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:34.216694 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.216670    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"]
Nov 03 04:08:34.217335 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:08:34.217309    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c524d7b_7651_4cca_b180_3bd053eeff2b.slice/crio-2be7f48ca4525b3fdf99b7782c00b6f12d00b45c8bc960d0a70b2adbe50493dc WatchSource:0}: Error finding container 2be7f48ca4525b3fdf99b7782c00b6f12d00b45c8bc960d0a70b2adbe50493dc: Status 404 returned error can't find the container with id 2be7f48ca4525b3fdf99b7782c00b6f12d00b45c8bc960d0a70b2adbe50493dc
Nov 03 04:08:34.219959 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.219929    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:34.220035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.219990    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:34.220035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.220020    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:34.345775 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.345747    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"]
Nov 03 04:08:34.346697 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:08:34.346673    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e843ff_8d4f_4276_9db2_83febf7e039b.slice/crio-d2a6b3e8ff85903aa0e6173ae22418e82253cd11c91823718f3f13c01b97ebc2 WatchSource:0}: Error finding container d2a6b3e8ff85903aa0e6173ae22418e82253cd11c91823718f3f13c01b97ebc2: Status 404 returned error can't find the container with id d2a6b3e8ff85903aa0e6173ae22418e82253cd11c91823718f3f13c01b97ebc2
Nov 03 04:08:34.349927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.349895    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:34.350006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.349958    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:34.350006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.349990    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:34.371295 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.371261    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" event={"ID":"4c524d7b-7651-4cca-b180-3bd053eeff2b","Type":"ContainerStarted","Data":"d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"}
Nov 03 04:08:34.371434 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.371327    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" event={"ID":"4c524d7b-7651-4cca-b180-3bd053eeff2b","Type":"ContainerStarted","Data":"2be7f48ca4525b3fdf99b7782c00b6f12d00b45c8bc960d0a70b2adbe50493dc"}
Nov 03 04:08:34.372932 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.372907    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" event={"ID":"e9e843ff-8d4f-4276-9db2-83febf7e039b","Type":"ContainerStarted","Data":"d2a6b3e8ff85903aa0e6173ae22418e82253cd11c91823718f3f13c01b97ebc2"}
Nov 03 04:08:34.494561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.494503    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podStartSLOduration=1.494485417 podStartE2EDuration="1.494485417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:34.493178132 +0000 UTC m=+4005.724070551" watchObservedRunningTime="2026-11-03 04:08:34.494485417 +0000 UTC m=+4005.725377838"
Nov 03 04:08:34.789598 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:34.789518    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:35.381615 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.381579    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" event={"ID":"e9e843ff-8d4f-4276-9db2-83febf7e039b","Type":"ContainerStarted","Data":"5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31"}
Nov 03 04:08:35.507210 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.507136    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podStartSLOduration=2.507116787 podStartE2EDuration="2.507116787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:35.50403036 +0000 UTC m=+4006.734922780" watchObservedRunningTime="2026-11-03 04:08:35.507116787 +0000 UTC m=+4006.738009209"
Nov 03 04:08:35.691456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.691346    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:35.795792 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.795763    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:35.958183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.958110    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:35.959560 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.959518    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:35.959710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:35.959599    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:36.459617 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.459541    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:36.460373 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.460351    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"]
Nov 03 04:08:36.497241 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.497209    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:36.500939 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.500913    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:08:36.501055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.500993    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.501611 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.501588    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"]
Nov 03 04:08:36.589654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.572940    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-eed598-openshift-default-dockercfg-9t6gt\""
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.607715    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.607923    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.607977    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.608289    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.608326    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.608374    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.608407    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8vn\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.609097    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.609528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.609285    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.650532 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.650493    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:36.710518 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710438    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710518 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710486    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710752 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710580    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710752 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710687    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710752 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710715    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710831    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710937 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710918    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.710975 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710959    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.711012 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.710990    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.711050 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.711024    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8vn\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.711094 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.711076    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.711463 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.711443    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.711746 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.711726    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.712125 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.712085    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.714732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.714708    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.715408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.715388    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.797384 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.797252    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.800856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.800831    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8vn\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn\") pod \"e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") " pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.823955 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.823928    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:36.959061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.959026    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:36.959218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:36.959089    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:37.290747 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.290709    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"]
Nov 03 04:08:37.292855 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:08:37.292829    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df20b96_a45e_43bb_b55d_b22ddd756896.slice/crio-79235ea6d90a9804699371ec9cc0bcbbc3f8b12a9e42e1b5645a28180295846c WatchSource:0}: Error finding container 79235ea6d90a9804699371ec9cc0bcbbc3f8b12a9e42e1b5645a28180295846c: Status 404 returned error can't find the container with id 79235ea6d90a9804699371ec9cc0bcbbc3f8b12a9e42e1b5645a28180295846c
Nov 03 04:08:37.295751 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.295717    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:37.295851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.295781    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:37.295851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.295815    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:08:37.391486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.391441    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:37.493593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.489474    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" event={"ID":"9df20b96-a45e-43bb-b55d-b22ddd756896","Type":"ContainerStarted","Data":"50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"}
Nov 03 04:08:37.494676 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.494643    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" event={"ID":"9df20b96-a45e-43bb-b55d-b22ddd756896","Type":"ContainerStarted","Data":"79235ea6d90a9804699371ec9cc0bcbbc3f8b12a9e42e1b5645a28180295846c"}
Nov 03 04:08:37.591428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.591327    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:37.611897 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.611849    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podStartSLOduration=1.611835813 podStartE2EDuration="1.611835813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:08:37.610737544 +0000 UTC m=+4008.841629992" watchObservedRunningTime="2026-11-03 04:08:37.611835813 +0000 UTC m=+4008.842728231"
Nov 03 04:08:37.735583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.735532    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:37.832687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.832646    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:37.958774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.958736    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:37.958933 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:37.958806    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:38.119774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:38.119744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:38.832724 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:38.832698    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:38.958959 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:38.958931    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:38.959143 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:38.958995    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:39.187904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:39.187870    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:39.340697 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:39.340665    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:39.500106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:39.500027    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:39.501031 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:39.501014    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:08:39.958614 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:39.958544    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:39.959018 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:39.958631    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:40.958249 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:40.958220    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:40.958483 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:40.958267    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:41.345881 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.345805    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:08:41.345881 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.345878    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:08:41.366270 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.366240    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:41.539921 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.539892    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:41.682991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.682942    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:41.958723 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.958646    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rbh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 04:08:41.958723 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:41.958705    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 04:08:42.992949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:42.992917    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:43.209149 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:43.209115    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:43.496075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:43.496022    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:43.739610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:43.738101    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:43.739610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:43.739477    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:43.740419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:43.740396    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:08:45.539522 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:45.539492    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:45.711973 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:45.711919    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:48.330387 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:48.330353    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:50.986302 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:08:50.986261    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:08:53.221982 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:53.221945    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:08:58.032283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:08:58.032246    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:02.850361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:02.850330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:05.927683 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:05.927649    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:09:07.731236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:07.731200    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:11.078864 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:11.078827    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:12.627374 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:12.627346    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:17.545987 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:17.545955    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:20.987366 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:09:20.987332    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:09:22.350970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:22.350932    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:22.516093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:22.515997    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:22.659149 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:22.659115    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:27.283520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:27.283485    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:27.444342 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:27.444308    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:27.600752 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:27.600649    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:32.322886 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:32.322850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:33.874483 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:33.874452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:37.058173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:37.058138    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:37.216902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:37.216867    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:37.363802 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:37.363774    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:41.883790 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:41.883754    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:42.036028 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:42.035965    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:44.414363 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:44.414328    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:09:46.973600 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:46.973530    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:50.994483 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:09:50.994444    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:09:51.834225 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:51.834150    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:54.080602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:54.080568    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:54.260842 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:54.260806    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:09:56.668843 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:09:56.668815    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:01.274615 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.274580    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"]
Nov 03 04:10:01.278958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.278940    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.333853 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.333820    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-934bd4-serving-cert\""
Nov 03 04:10:01.337476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.337453    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"]
Nov 03 04:10:01.356561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.356505    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.356715 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.356674    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.356917 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.356895    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp6js\" (UniqueName: \"kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.356991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.356954    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.458680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.458650    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.458845 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.458803    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp6js\" (UniqueName: \"kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.458901 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.458848    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.458957 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.458900    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.459016 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:10:01.459000    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-serving-cert: secret "maas-api-e2e-inf-a-934bd4-serving-cert" not found
Nov 03 04:10:01.459071 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:10:01.459058    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls podName:083935e7-714c-4880-832b-0b1da75fe2f8 nodeName:}" failed. No retries permitted until 2026-11-03 04:10:01.959041282 +0000 UTC m=+4093.189933682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls") pod "maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" (UID: "083935e7-714c-4880-832b-0b1da75fe2f8") : secret "maas-api-e2e-inf-a-934bd4-serving-cert" not found
Nov 03 04:10:01.459518 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.459492    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.461415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.461391    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.472604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.472491    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:01.550746 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.549315    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp6js\" (UniqueName: \"kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.639812 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.639772    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:01.790606 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.790508    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:01.977226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.977184    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:01.979864 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:01.979839    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\") pod \"maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:02.192714 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:02.192677    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:02.571127 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:02.571103    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"]
Nov 03 04:10:02.571749 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:10:02.571721    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083935e7_714c_4880_832b_0b1da75fe2f8.slice/crio-589e4d31b6dbcef49c9375679133cf8a740d0c9219199bfe0e840b648bc2601d WatchSource:0}: Error finding container 589e4d31b6dbcef49c9375679133cf8a740d0c9219199bfe0e840b648bc2601d: Status 404 returned error can't find the container with id 589e4d31b6dbcef49c9375679133cf8a740d0c9219199bfe0e840b648bc2601d
Nov 03 04:10:02.573266 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:02.573248    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:10:03.095091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:03.095045    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" event={"ID":"083935e7-714c-4880-832b-0b1da75fe2f8","Type":"ContainerStarted","Data":"589e4d31b6dbcef49c9375679133cf8a740d0c9219199bfe0e840b648bc2601d"}
Nov 03 04:10:04.104336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:04.104291    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" event={"ID":"083935e7-714c-4880-832b-0b1da75fe2f8","Type":"ContainerStarted","Data":"a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a"}
Nov 03 04:10:04.104850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:04.104522    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:04.226905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:04.226853    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" podStartSLOduration=2.431984366 podStartE2EDuration="3.226839608s" totalImagesPullingTime="794.855242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:10:04.22475852 +0000 UTC m=+4095.455650917" watchObservedRunningTime="2026-11-03 04:10:04.226839608 +0000 UTC m=+4095.457732076"
Nov 03 04:10:06.531344 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:06.531299    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:10.116753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:10.116728    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:10:11.307959 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:11.307927    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:11.455712 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:11.455683    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:11.602030 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:11.601954    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:16.329272 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:16.329202    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:18.739243 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:18.739211    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:10:21.015124 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:10:21.015063    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:10:21.217247 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:21.217213    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:26.044376 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:26.044346    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:29.341925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:29.341885    5515 generic.go:356] "Generic (PLEG): container finished" podID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerID="d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd" exitCode=0
Nov 03 04:10:29.342413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:29.341965    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" event={"ID":"4c524d7b-7651-4cca-b180-3bd053eeff2b","Type":"ContainerDied","Data":"d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"}
Nov 03 04:10:29.342492 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:29.342473    5515 scope.go:122] "RemoveContainer" containerID="d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"
Nov 03 04:10:29.344197 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:29.344168    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:10:29.344277 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:29.344249    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:10:29.344319 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:29.344300    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:10:30.350488 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:30.350451    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" event={"ID":"4c524d7b-7651-4cca-b180-3bd053eeff2b","Type":"ContainerStarted","Data":"39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3"}
Nov 03 04:10:30.789747 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:30.789708    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:10:30.789937 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:30.789911    5515 patch_prober.go:29] interesting pod/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 04:10:30.789992 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:30.789965    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 04:10:30.931463 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:30.931432    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:31.789192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:31.789140    5515 patch_prober.go:29] interesting pod/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 04:10:31.789794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:31.789207    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 04:10:32.789493 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:32.789457    5515 patch_prober.go:29] interesting pod/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 04:10:32.789935 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:32.789526    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 04:10:33.789638 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:33.789604    5515 patch_prober.go:29] interesting pod/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 04:10:33.790037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:33.789655    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 04:10:34.789260 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:34.789214    5515 patch_prober.go:29] interesting pod/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 03 04:10:34.789479 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:34.789300    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 03 04:10:35.802010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:35.801978    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:10:35.802728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:35.802694    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:10:35.803604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:35.803585    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:10:35.821638 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:35.821604    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:40.647622 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:40.647587    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:45.453502 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:45.453452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:50.338489 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:50.338441    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:10:50.985092 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:10:50.985056    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:10:53.237116 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:53.237032    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:10:55.181338 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:10:55.181002    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:00.058914 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:00.058871    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:02.211030 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:02.210999    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:02.689866 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:02.689816    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:03.507827 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:03.463833    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:03.626991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:03.626954    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:04.703603 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:04.696501    5515 generic.go:356] "Generic (PLEG): container finished" podID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerID="50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5" exitCode=0
Nov 03 04:11:04.703603 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:04.696849    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" event={"ID":"9df20b96-a45e-43bb-b55d-b22ddd756896","Type":"ContainerDied","Data":"50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"}
Nov 03 04:11:04.703603 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:04.697506    5515 scope.go:122] "RemoveContainer" containerID="50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"
Nov 03 04:11:04.733576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:04.727377    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:11:04.747175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:04.747120    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:11:04.747381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:04.747226    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:11:05.406162 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:05.406122    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:05.707461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:05.707376    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" event={"ID":"9df20b96-a45e-43bb-b55d-b22ddd756896","Type":"ContainerStarted","Data":"60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404"}
Nov 03 04:11:05.824959 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:05.824932    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:11:05.825918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:05.825897    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 04:11:05.825970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:05.825953    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 04:11:06.824798 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:06.824766    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 04:11:06.825196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:06.824843    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 04:11:07.672236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:07.672187    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:07.825257 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:07.825224    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 04:11:07.825684 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:07.825275    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 04:11:07.839434 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:07.839407    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:07.988863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:07.988776    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:08.824789 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:08.824750    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 04:11:08.824989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:08.824805    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 04:11:09.500813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:09.500783    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:11:09.824709 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:09.824615    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 04:11:09.824709 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:09.824672    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 04:11:10.411037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:10.411003    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:10.623080 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:10.623048    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:10.824780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:10.824685    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 04:11:10.824780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:10.824746    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 04:11:10.970317 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:10.970282    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:11.154442 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:11.154408    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:11.803615 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:11.803582    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:11.829346 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:11.829318    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:11:11.830157 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:11.830138    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:11:14.285136 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:14.285101    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:15.716963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:15.716920    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:16.711450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:16.711425    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:16.858636 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:16.858600    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:19.841221 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:19.840522    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"]
Nov 03 04:11:19.949933 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:19.949898    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"]
Nov 03 04:11:20.199805 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:20.199759    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:20.351154 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:20.351120    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:20.876464 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:20.876418    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:21.036904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:21.036778    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:21.047752 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:11:21.047504    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:11:22.734113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:22.734069    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:24.841253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:24.841217    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr" podUID="0de58841-d1a2-4027-b005-65cb7506c1e3" containerName="payload-processing" containerID="cri-o://9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5" gracePeriod=25
Nov 03 04:11:24.909489 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:11:24.909448    5515 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de58841_d1a2_4027_b005_65cb7506c1e3.slice/crio-9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de58841_d1a2_4027_b005_65cb7506c1e3.slice/crio-conmon-9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 04:11:24.950858 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:24.950806    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk" podUID="9fd73f79-7c61-4766-b7bd-4e8393d612c5" containerName="payload-pre-processing" containerID="cri-o://d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832" gracePeriod=25
Nov 03 04:11:25.112278 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.111846    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:11:25.150256 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.143624    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v\" (UniqueName: \"kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v\") pod \"0de58841-d1a2-4027-b005-65cb7506c1e3\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") "
Nov 03 04:11:25.150256 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.143683    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume\") pod \"0de58841-d1a2-4027-b005-65cb7506c1e3\" (UID: \"0de58841-d1a2-4027-b005-65cb7506c1e3\") "
Nov 03 04:11:25.150256 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.144825    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume" pod "0de58841-d1a2-4027-b005-65cb7506c1e3" (UID: "0de58841-d1a2-4027-b005-65cb7506c1e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:11:25.150899 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.150866    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v" pod "0de58841-d1a2-4027-b005-65cb7506c1e3" (UID: "0de58841-d1a2-4027-b005-65cb7506c1e3"). InnerVolumeSpecName "kube-api-access-t6s2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:11:25.219179 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.219154    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:11:25.245091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.245055    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth\" (UniqueName: \"kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth\") pod \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") "
Nov 03 04:11:25.245244 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.245218    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume\") pod \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\" (UID: \"9fd73f79-7c61-4766-b7bd-4e8393d612c5\") "
Nov 03 04:11:25.245848 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.245799    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6s2v\" (UniqueName: \"kubernetes.io/projected/0de58841-d1a2-4027-b005-65cb7506c1e3-kube-api-access-t6s2v\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:11:25.245995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.245925    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0de58841-d1a2-4027-b005-65cb7506c1e3-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:11:25.245995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.245814    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume" pod "9fd73f79-7c61-4766-b7bd-4e8393d612c5" (UID: "9fd73f79-7c61-4766-b7bd-4e8393d612c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:11:25.247397 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.247366    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth" pod "9fd73f79-7c61-4766-b7bd-4e8393d612c5" (UID: "9fd73f79-7c61-4766-b7bd-4e8393d612c5"). InnerVolumeSpecName "kube-api-access-czcth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:11:25.347234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.347196    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd73f79-7c61-4766-b7bd-4e8393d612c5-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:11:25.347234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.347224    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czcth\" (UniqueName: \"kubernetes.io/projected/9fd73f79-7c61-4766-b7bd-4e8393d612c5-kube-api-access-czcth\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:11:25.941411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.941379    5515 generic.go:356] "Generic (PLEG): container finished" podID="9fd73f79-7c61-4766-b7bd-4e8393d612c5" containerID="d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832" exitCode=0
Nov 03 04:11:25.941822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.941423    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"
Nov 03 04:11:25.941822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.941468    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk" event={"ID":"9fd73f79-7c61-4766-b7bd-4e8393d612c5","Type":"ContainerDied","Data":"d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832"}
Nov 03 04:11:25.941822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.941516    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk" event={"ID":"9fd73f79-7c61-4766-b7bd-4e8393d612c5","Type":"ContainerDied","Data":"56e2743e3451d23a5a73d7f032555975f4ebde70ce890fd338f100a9a1ecea95"}
Nov 03 04:11:25.941822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.941566    5515 scope.go:122] "RemoveContainer" containerID="d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832"
Nov 03 04:11:25.942985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.942963    5515 generic.go:356] "Generic (PLEG): container finished" podID="0de58841-d1a2-4027-b005-65cb7506c1e3" containerID="9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5" exitCode=0
Nov 03 04:11:25.943106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.943019    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"
Nov 03 04:11:25.943106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.943051    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr" event={"ID":"0de58841-d1a2-4027-b005-65cb7506c1e3","Type":"ContainerDied","Data":"9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5"}
Nov 03 04:11:25.943214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.943125    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr" event={"ID":"0de58841-d1a2-4027-b005-65cb7506c1e3","Type":"ContainerDied","Data":"39201316644b64552d117607334c7b68428424068e101fe51bd321c1843a29f5"}
Nov 03 04:11:25.952234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.952213    5515 scope.go:122] "RemoveContainer" containerID="d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832"
Nov 03 04:11:25.952527 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:11:25.952505    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832\": container with ID starting with d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832 not found: ID does not exist" containerID="d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832"
Nov 03 04:11:25.952682 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.952535    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832"} err="failed to get container status \"d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832\": rpc error: code = NotFound desc = could not find container \"d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832\": container with ID starting with d6a20f0e70991f085fecc21cd055cf86e68267ed7b0ea23ce9712803dea3b832 not found: ID does not exist"
Nov 03 04:11:25.952740 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.952687    5515 scope.go:122] "RemoveContainer" containerID="9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5"
Nov 03 04:11:25.961992 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.961976    5515 scope.go:122] "RemoveContainer" containerID="9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5"
Nov 03 04:11:25.962259 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:11:25.962241    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5\": container with ID starting with 9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5 not found: ID does not exist" containerID="9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5"
Nov 03 04:11:25.962307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:25.962265    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5"} err="failed to get container status \"9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5\": rpc error: code = NotFound desc = could not find container \"9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5\": container with ID starting with 9099784bac34f17f20e5cad491c29059a4db0bd38772ae303fdaa9b2061cc1e5 not found: ID does not exist"
Nov 03 04:11:26.108118 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.108093    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"]
Nov 03 04:11:26.118732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.118703    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4fa79-56bf984df9-5gdmk"]
Nov 03 04:11:26.136094 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.136067    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:26.225353 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.225318    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"]
Nov 03 04:11:26.228036 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.228012    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4fa79-59c69ff65f-pqcjr"]
Nov 03 04:11:26.895232 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.895191    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de58841-d1a2-4027-b005-65cb7506c1e3" path="/var/lib/kubelet/pods/0de58841-d1a2-4027-b005-65cb7506c1e3/volumes"
Nov 03 04:11:26.895733 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:26.895711    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd73f79-7c61-4766-b7bd-4e8393d612c5" path="/var/lib/kubelet/pods/9fd73f79-7c61-4766-b7bd-4e8393d612c5/volumes"
Nov 03 04:11:27.147187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:27.147109    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:30.966817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:30.966782    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:35.796810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:35.796776    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:36.909780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:36.909744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:40.115992 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:40.115953    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:40.624842 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:40.624810    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:42.425218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:42.425188    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:42.574561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:42.574512    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:43.306466 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:43.306435    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:45.440892 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:45.440860    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:45.585485 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:45.585452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:45.754622 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:45.753892    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:47.190177 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:47.190145    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:47.789762 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:47.789726    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:47.934871 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:47.934836    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:49.036370 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:49.036333    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:50.182252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:50.182218    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:50.304698 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:50.304665    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:11:50.307018 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:50.306994    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:11:50.332140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:50.332120    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:11:50.334503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:50.334482    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:11:50.984998 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:11:50.984954    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:11:53.764844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:53.764790    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:55.048263 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:55.048205    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:11:59.940419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:11:59.940354    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:04.765774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:04.765739    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:06.907341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:06.907290    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:07.054431 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:07.054375    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:09.589470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:09.589438    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:12.148688 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:12.148621    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:14.403889 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:14.403796    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:19.220315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:19.220251    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:20.984227 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:12:20.984186    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:12:24.098504 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:24.098423    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:28.944160 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:28.944125    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:33.199163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:33.199129    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:35.039388 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:35.039350    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:39.968043 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:39.967998    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:40.178385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:40.178335    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:40.324563 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:40.324454    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:41.413839 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:41.413807    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:41.570620 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:41.570586    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:42.131925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:42.131888    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:42.861440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:42.861393    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:44.739906 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:44.739874    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:45.892848 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:45.892810    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:46.051945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:46.051915    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:46.201851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:46.201769    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:46.622446 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:46.622411    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:46.778663 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:46.778603    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:47.320151 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:47.320117    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:49.692320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:49.692282    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:12:49.752342 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:49.752309    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:50.526763 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:50.525444    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:50.817035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:50.816946    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:50.984772 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:12:50.984733    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:12:52.205789 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:52.205748    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:54.196090 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:54.196056    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:54.347293 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:54.347267    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:54.597985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:54.597912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:54.786893 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:54.786864    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:54.987100 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:54.987065    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:57.523075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:57.523037    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:58.536772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:58.536741    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:12:58.744603 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:12:58.738828    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:03.027120 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:03.027079    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:08.291781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:08.291748    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:12.571955 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:12.571912    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:13.878573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:13.878072    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:14.090575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:14.090359    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:13:14.090575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:14.090434    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:13:14.161623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:14.153845    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:13:14.161623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:14.153906    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:13:19.249205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.249170    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:19.396747 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.396711    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"]
Nov 03 04:13:19.397414 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.397398    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9fd73f79-7c61-4766-b7bd-4e8393d612c5" containerName="payload-pre-processing"
Nov 03 04:13:19.397460 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.397420    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd73f79-7c61-4766-b7bd-4e8393d612c5" containerName="payload-pre-processing"
Nov 03 04:13:19.397460 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.397440    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0de58841-d1a2-4027-b005-65cb7506c1e3" containerName="payload-processing"
Nov 03 04:13:19.397460 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.397445    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de58841-d1a2-4027-b005-65cb7506c1e3" containerName="payload-processing"
Nov 03 04:13:19.397601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.397591    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de58841-d1a2-4027-b005-65cb7506c1e3" containerName="payload-processing"
Nov 03 04:13:19.397638 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.397613    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd73f79-7c61-4766-b7bd-4e8393d612c5" containerName="payload-pre-processing"
Nov 03 04:13:19.400054 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.400034    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.453261 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.453230    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8540f6-openshift-default-dockercfg-pv8rg\""
Nov 03 04:13:19.490211 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490173    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdx5\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490255    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490353    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490404    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490658 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490431    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490658 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490540    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490658 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490605    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.490860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490827    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.491012 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.490900    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.509078 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.509016    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"]
Nov 03 04:13:19.592081 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592049    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592141    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592184    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592203    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592439 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592295    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592439 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592335    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592439 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592402    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592633 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592456    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592633 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592500    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdx5\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592770 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592747    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.592846 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592773    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.593011 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592978    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.593011 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.592987    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.593183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.593023    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.595037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.595012    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.595151 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.595134    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.657653 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.657624    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.657833 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.657802    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdx5\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5\") pod \"e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") " pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:19.712905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:19.712878    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:20.146672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:20.146647    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"]
Nov 03 04:13:20.148187 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:13:20.148162    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c637e7_674e_4ff9_840e_586143a6a614.slice/crio-f3d981667b60fe5f2c810eddcac9e0a32f6423b07ff4ea9a8b5c87d189bbd0a0 WatchSource:0}: Error finding container f3d981667b60fe5f2c810eddcac9e0a32f6423b07ff4ea9a8b5c87d189bbd0a0: Status 404 returned error can't find the container with id f3d981667b60fe5f2c810eddcac9e0a32f6423b07ff4ea9a8b5c87d189bbd0a0
Nov 03 04:13:20.150884 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:20.150848    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:13:20.150985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:20.150917    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:13:20.150985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:20.150947    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:13:20.836587 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:20.836536    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" event={"ID":"e2c637e7-674e-4ff9-840e-586143a6a614","Type":"ContainerStarted","Data":"4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d"}
Nov 03 04:13:20.837005 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:20.836613    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" event={"ID":"e2c637e7-674e-4ff9-840e-586143a6a614","Type":"ContainerStarted","Data":"f3d981667b60fe5f2c810eddcac9e0a32f6423b07ff4ea9a8b5c87d189bbd0a0"}
Nov 03 04:13:21.010482 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:13:21.010447    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:13:21.013913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:21.013870    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podStartSLOduration=2.013856249 podStartE2EDuration="2.013856249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:13:21.01245542 +0000 UTC m=+4292.243347839" watchObservedRunningTime="2026-11-03 04:13:21.013856249 +0000 UTC m=+4292.244748666"
Nov 03 04:13:21.713908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:21.713877    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:21.715734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:21.715712    5515 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 03 04:13:21.715808 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:21.715758    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 03 04:13:22.713569 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:22.713512    5515 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 03 04:13:22.714063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:22.713613    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 03 04:13:23.713842 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:23.713806    5515 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 03 04:13:23.714259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:23.713861    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 03 04:13:24.133764 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:24.133715    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:24.713855 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:24.713824    5515 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 03 04:13:24.714257 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:24.713891    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 03 04:13:25.713533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:25.713502    5515 patch_prober.go:29] interesting pod/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 03 04:13:25.713745 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:25.713588    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 03 04:13:26.717914 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:26.717879    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:26.718310 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:26.718292    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:26.719064 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:26.719043    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:13:28.966913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:28.966875    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:33.842177 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:33.842143    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:36.307997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:36.307953    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:36.457453 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:36.457420    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:37.038062 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:37.038026    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:13:38.760737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:38.760705    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:43.632788 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:43.632754    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:48.514625 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:48.514584    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:50.987000 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:13:50.986957    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:13:53.385600 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:53.385489    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:13:58.224000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:13:58.223967    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:00.996160 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:00.996127    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:03.125986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:03.125947    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:07.390355 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:07.390317    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:07.843394 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:07.843301    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:08.021441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:08.021415    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:12.902731 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:12.902695    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:13.116952 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:13.116920    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:13.405734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:13.405699    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:13.551063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:13.551032    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:13.800631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:13.800471    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:18.232496 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:18.232462    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:18.475458 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:18.475423    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:18.619164 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:18.619131    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:20.151035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:20.150991    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:20.984778 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:20.984744    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:14:23.427794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:23.427762    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:28.324358 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:28.324323    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:32.465984 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.465950    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"]
Nov 03 04:14:32.466601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.466199    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" podUID="083935e7-714c-4880-832b-0b1da75fe2f8" containerName="maas-api" containerID="cri-o://a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a" gracePeriod=30
Nov 03 04:14:32.572274 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.572238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:32.821287 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.821260    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:14:32.850813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.850788    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca\") pod \"083935e7-714c-4880-832b-0b1da75fe2f8\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") "
Nov 03 04:14:32.850928 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.850820    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs\") pod \"083935e7-714c-4880-832b-0b1da75fe2f8\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") "
Nov 03 04:14:32.850928 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.850910    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\") pod \"083935e7-714c-4880-832b-0b1da75fe2f8\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") "
Nov 03 04:14:32.851032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.851014    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js\" (UniqueName: \"kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js\") pod \"083935e7-714c-4880-832b-0b1da75fe2f8\" (UID: \"083935e7-714c-4880-832b-0b1da75fe2f8\") "
Nov 03 04:14:32.851123 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.851101    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca" pod "083935e7-714c-4880-832b-0b1da75fe2f8" (UID: "083935e7-714c-4880-832b-0b1da75fe2f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:14:32.851584 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.851526    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083935e7-714c-4880-832b-0b1da75fe2f8-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:32.853123 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.853094    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls" pod "083935e7-714c-4880-832b-0b1da75fe2f8" (UID: "083935e7-714c-4880-832b-0b1da75fe2f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:14:32.853211 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.853132    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs" pod "083935e7-714c-4880-832b-0b1da75fe2f8" (UID: "083935e7-714c-4880-832b-0b1da75fe2f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:14:32.853211 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.853203    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js" pod "083935e7-714c-4880-832b-0b1da75fe2f8" (UID: "083935e7-714c-4880-832b-0b1da75fe2f8"). InnerVolumeSpecName "kube-api-access-pp6js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:14:32.959757 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.959719    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:32.959757 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.959759    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083935e7-714c-4880-832b-0b1da75fe2f8-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:32.960029 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:32.959781    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp6js\" (UniqueName: \"kubernetes.io/projected/083935e7-714c-4880-832b-0b1da75fe2f8-kube-api-access-pp6js\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:33.492575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.487604    5515 generic.go:356] "Generic (PLEG): container finished" podID="083935e7-714c-4880-832b-0b1da75fe2f8" containerID="a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a" exitCode=0
Nov 03 04:14:33.492575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.487938    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" event={"ID":"083935e7-714c-4880-832b-0b1da75fe2f8","Type":"ContainerDied","Data":"a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a"}
Nov 03 04:14:33.492575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.488047    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh" event={"ID":"083935e7-714c-4880-832b-0b1da75fe2f8","Type":"ContainerDied","Data":"589e4d31b6dbcef49c9375679133cf8a740d0c9219199bfe0e840b648bc2601d"}
Nov 03 04:14:33.492575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.488079    5515 scope.go:122] "RemoveContainer" containerID="a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a"
Nov 03 04:14:33.492575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.488288    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"
Nov 03 04:14:33.551523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.551498    5515 scope.go:122] "RemoveContainer" containerID="a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a"
Nov 03 04:14:33.552047 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:33.552029    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a\": container with ID starting with a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a not found: ID does not exist" containerID="a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a"
Nov 03 04:14:33.552185 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.552165    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a"} err="failed to get container status \"a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a\": rpc error: code = NotFound desc = could not find container \"a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a\": container with ID starting with a3c334b3425a0fe277c249cf24ca12e5a9401cb665bda1f2db2fd393f165f88a not found: ID does not exist"
Nov 03 04:14:33.659874 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.659842    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"]
Nov 03 04:14:33.709236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:33.709204    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-934bd4-7c7b479dd5-jpfjh"]
Nov 03 04:14:34.273219 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:34.273179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:34.419945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:34.419917    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:34.923300 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:34.922501    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083935e7-714c-4880-832b-0b1da75fe2f8" path="/var/lib/kubelet/pods/083935e7-714c-4880-832b-0b1da75fe2f8/volumes"
Nov 03 04:14:34.961915 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:34.961853    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:34.966615 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:34.966589    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"]
Nov 03 04:14:34.967398 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:34.967376    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" containerID="cri-o://39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3" gracePeriod=30
Nov 03 04:14:35.193203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:35.193118    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:14:35.804904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:35.804863    5515 patch_prober.go:29] interesting pod/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 04:14:35.805081 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:35.804942    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 04:14:37.612873 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:37.612843    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:37.768312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:37.768279    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:38.649942 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:38.649906    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:38.796863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:38.796830    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:38.977187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:38.977110    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:40.242925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.242871    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:14:40.329765 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.329732    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.329956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.329790    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.329956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.329828    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.329956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.329850    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.329956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.329888    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.330208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330014    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:40.330208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330049    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.330208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330071    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.330208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330172    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.330433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330224    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token\") pod \"4c524d7b-7651-4cca-b180-3bd053eeff2b\" (UID: \"4c524d7b-7651-4cca-b180-3bd053eeff2b\") "
Nov 03 04:14:40.330433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330171    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:40.330433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330254    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:14:40.330433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330384    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:40.330433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.330416    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:40.331074 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.331047    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.331167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.331080    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c524d7b-7651-4cca-b180-3bd053eeff2b-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.331167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.331096    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.331167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.331111    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.331167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.331126    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.332451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.332428    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:14:40.332451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.332440    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:40.332618 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.332450    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "kube-api-access-gqqhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:14:40.332618 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.332477    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token" pod "4c524d7b-7651-4cca-b180-3bd053eeff2b" (UID: "4c524d7b-7651-4cca-b180-3bd053eeff2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:14:40.432721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.432690    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.432721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.432724    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.432908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.432739    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqhv\" (UniqueName: \"kubernetes.io/projected/4c524d7b-7651-4cca-b180-3bd053eeff2b-kube-api-access-gqqhv\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.432908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.432754    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c524d7b-7651-4cca-b180-3bd053eeff2b-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:40.536132 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.536047    5515 generic.go:356] "Generic (PLEG): container finished" podID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerID="39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3" exitCode=0
Nov 03 04:14:40.536132 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.536100    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"
Nov 03 04:14:40.536336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.536099    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" event={"ID":"4c524d7b-7651-4cca-b180-3bd053eeff2b","Type":"ContainerDied","Data":"39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3"}
Nov 03 04:14:40.536336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.536220    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx" event={"ID":"4c524d7b-7651-4cca-b180-3bd053eeff2b","Type":"ContainerDied","Data":"2be7f48ca4525b3fdf99b7782c00b6f12d00b45c8bc960d0a70b2adbe50493dc"}
Nov 03 04:14:40.536336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.536244    5515 scope.go:122] "RemoveContainer" containerID="39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3"
Nov 03 04:14:40.547444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.547426    5515 scope.go:122] "RemoveContainer" containerID="d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"
Nov 03 04:14:40.556533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.556514    5515 scope.go:122] "RemoveContainer" containerID="39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3"
Nov 03 04:14:40.556837 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:40.556816    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3\": container with ID starting with 39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3 not found: ID does not exist" containerID="39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3"
Nov 03 04:14:40.556913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.556852    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3"} err="failed to get container status \"39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3\": rpc error: code = NotFound desc = could not find container \"39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3\": container with ID starting with 39153a63eba74e24a9fcba26dd6250bb670163c33713b3719a36ca2dc72c89b3 not found: ID does not exist"
Nov 03 04:14:40.556913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.556876    5515 scope.go:122] "RemoveContainer" containerID="d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"
Nov 03 04:14:40.557137 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:40.557120    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd\": container with ID starting with d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd not found: ID does not exist" containerID="d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"
Nov 03 04:14:40.557179 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.557146    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd"} err="failed to get container status \"d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd\": rpc error: code = NotFound desc = could not find container \"d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd\": container with ID starting with d4fa8005448d63f5e6fcacba283617ee99dcae275900a3f554ac97db9556dffd not found: ID does not exist"
Nov 03 04:14:40.677139 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.677106    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:40.704729 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.704697    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"]
Nov 03 04:14:40.708943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.708923    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-934bd4-openshift-default-5b5d5d8479-kphgx"]
Nov 03 04:14:40.824780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.824691    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:40.896350 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:40.896321    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" path="/var/lib/kubelet/pods/4c524d7b-7651-4cca-b180-3bd053eeff2b/volumes"
Nov 03 04:14:41.180259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:41.180221    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"]
Nov 03 04:14:41.180540 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:41.180510    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" containerID="cri-o://60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404" gracePeriod=30
Nov 03 04:14:41.218885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:41.218858    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:41.832956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:41.832923    5515 patch_prober.go:29] interesting pod/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 04:14:41.833319 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:41.832994    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 04:14:44.149516 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:44.149483    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:44.788601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:44.788569    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:45.095623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:45.091631    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:45.308328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:45.308303    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:45.455145 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:45.455112    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:46.027734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.027704    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:46.457817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.457794    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:14:46.608218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.608191    5515 generic.go:356] "Generic (PLEG): container finished" podID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerID="60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404" exitCode=0
Nov 03 04:14:46.608413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.608236    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" event={"ID":"9df20b96-a45e-43bb-b55d-b22ddd756896","Type":"ContainerDied","Data":"60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404"}
Nov 03 04:14:46.608413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.608253    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"
Nov 03 04:14:46.608413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.608284    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l" event={"ID":"9df20b96-a45e-43bb-b55d-b22ddd756896","Type":"ContainerDied","Data":"79235ea6d90a9804699371ec9cc0bcbbc3f8b12a9e42e1b5645a28180295846c"}
Nov 03 04:14:46.608413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.608309    5515 scope.go:122] "RemoveContainer" containerID="60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404"
Nov 03 04:14:46.618391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.618364    5515 scope.go:122] "RemoveContainer" containerID="50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"
Nov 03 04:14:46.626876 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.626850    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.626988 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.626920    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.626988 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.626974    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627095 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627039    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627095 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627066    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627098    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627168    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627244    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:46.627371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627339    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627412 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627378    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn\") pod \"9df20b96-a45e-43bb-b55d-b22ddd756896\" (UID: \"9df20b96-a45e-43bb-b55d-b22ddd756896\") "
Nov 03 04:14:46.627473 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627425    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:46.627979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627565    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:46.627979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627654    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:14:46.627979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.627778    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:46.628170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.628000    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.628170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.628036    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df20b96-a45e-43bb-b55d-b22ddd756896-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.628170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.628051    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.628170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.628065    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.628170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.628078    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.630472 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.630451    5515 scope.go:122] "RemoveContainer" containerID="60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404"
Nov 03 04:14:46.630743 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.630720    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:14:46.630851 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:46.630824    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404\": container with ID starting with 60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404 not found: ID does not exist" containerID="60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404"
Nov 03 04:14:46.630921 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.630847    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "kube-api-access-cb8vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:14:46.630921 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.630858    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404"} err="failed to get container status \"60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404\": rpc error: code = NotFound desc = could not find container \"60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404\": container with ID starting with 60bc403a21c61072044a0aafc3fff076f9ea000d67f2024802f78ec54b176404 not found: ID does not exist"
Nov 03 04:14:46.630921 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.630883    5515 scope.go:122] "RemoveContainer" containerID="50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"
Nov 03 04:14:46.631104 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.630998    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:14:46.631239 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:46.631214    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5\": container with ID starting with 50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5 not found: ID does not exist" containerID="50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"
Nov 03 04:14:46.631291 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.631249    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5"} err="failed to get container status \"50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5\": rpc error: code = NotFound desc = could not find container \"50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5\": container with ID starting with 50130e25d2fb4c7cce745de2b631d741d93c0ff273f8e4cf3319be1c0a77efb5 not found: ID does not exist"
Nov 03 04:14:46.631599 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.631577    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo" pod "9df20b96-a45e-43bb-b55d-b22ddd756896" (UID: "9df20b96-a45e-43bb-b55d-b22ddd756896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:14:46.729259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.729227    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.729259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.729254    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8vn\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-kube-api-access-cb8vn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.729259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.729265    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:46.729509 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:46.729273    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df20b96-a45e-43bb-b55d-b22ddd756896-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:14:47.057821 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:47.057256    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"]
Nov 03 04:14:47.058906 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:47.058855    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-eed598-openshift-default-c64cb594c-rt22l"]
Nov 03 04:14:48.710467 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:48.710433    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:48.894919 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:48.894890    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" path="/var/lib/kubelet/pods/9df20b96-a45e-43bb-b55d-b22ddd756896/volumes"
Nov 03 04:14:49.872237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:49.872174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:50.054306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:50.054262    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:50.213837 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:50.213753    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:50.984084 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:14:50.984047    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:14:51.380467 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:51.380431    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:53.121754 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:53.121698    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:53.304584 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:53.296083    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:54.897603 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:54.897567    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:14:56.505931 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:14:56.505897    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:00.215389 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.215354    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"]
Nov 03 04:15:00.215995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.215976    5515 cpu_manager.go:414] "containerMap: removing container" podUID="083935e7-714c-4880-832b-0b1da75fe2f8" containerName="maas-api"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.215996    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083935e7-714c-4880-832b-0b1da75fe2f8" containerName="maas-api"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216024    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216030    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216050    5515 cpu_manager.go:414] "containerMap: removing container" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216055    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216064    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy"
Nov 03 04:15:00.216076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216068    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy"
Nov 03 04:15:00.216307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216179    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy"
Nov 03 04:15:00.216307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216188    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy"
Nov 03 04:15:00.216307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216194    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy"
Nov 03 04:15:00.216307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.216203    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="083935e7-714c-4880-832b-0b1da75fe2f8" containerName="maas-api"
Nov 03 04:15:00.220976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.220959    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"]
Nov 03 04:15:00.221092 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.221072    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.221621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.221606    5515 cpu_manager.go:414] "containerMap: removing container" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy"
Nov 03 04:15:00.221680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.221622    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df20b96-a45e-43bb-b55d-b22ddd756896" containerName="istio-proxy"
Nov 03 04:15:00.221814 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.221803    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c524d7b-7651-4cca-b180-3bd053eeff2b" containerName="istio-proxy"
Nov 03 04:15:00.225386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.225366    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"]
Nov 03 04:15:00.225526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.225483    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.229271 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.229252    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.265676 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.265642    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"]
Nov 03 04:15:00.269720 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.269697    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"]
Nov 03 04:15:00.269860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.269814    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.273962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.273516    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"]
Nov 03 04:15:00.274286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.274261    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.277951 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.277932    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.346784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.346664    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"]
Nov 03 04:15:00.349264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.349239    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6kqpw\""
Nov 03 04:15:00.349264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.349242    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"]
Nov 03 04:15:00.351900 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.351876    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"]
Nov 03 04:15:00.372781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.372743    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.373028 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.372862    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.373028 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.372940    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.373148 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373056    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q722h\" (UniqueName: \"kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.373148 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373138    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.373251 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373179    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k77\" (UniqueName: \"kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.373251 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373212    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czhk\" (UniqueName: \"kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk\") pod \"maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.373362 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373301    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.373362 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373325    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.373529 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373498    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nb46\" (UniqueName: \"kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46\") pod \"maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.373693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373588    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8sr\" (UniqueName: \"kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.373744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.373696    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qnl\" (UniqueName: \"kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.390076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.390029    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"]
Nov 03 04:15:00.411055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.410946    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"]
Nov 03 04:15:00.414287 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.414253    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"]
Nov 03 04:15:00.474840 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.474754    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q722h\" (UniqueName: \"kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.474840 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.474816    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.474852    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k77\" (UniqueName: \"kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.474876    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czhk\" (UniqueName: \"kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk\") pod \"maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.474920    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.474950    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475034    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nb46\" (UniqueName: \"kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46\") pod \"maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475066    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8sr\" (UniqueName: \"kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.475188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475120    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qnl\" (UniqueName: \"kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.475796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475211    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.475796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475283    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.475796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475322    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.475796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475740    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.476031 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475896    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.476031 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.475909    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.476153 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.476095    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.476395 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.476371    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.476470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.476375    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.548437 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.548399    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q722h\" (UniqueName: \"kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h\") pod \"maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.548941 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.548917    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8sr\" (UniqueName: \"kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.549580 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.549519    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k77\" (UniqueName: \"kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77\") pod \"maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.549804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.549782    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qnl\" (UniqueName: \"kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.550577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.550531    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:00.556411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.556376    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nb46\" (UniqueName: \"kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46\") pod \"maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.559230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.559207    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czhk\" (UniqueName: \"kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk\") pod \"maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.585742 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.585712    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:00.598079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.598042    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:00.602505 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.602481    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:00.836934 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.836904    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:00.844918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.844897    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:00.938586 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:00.938560    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"]
Nov 03 04:15:00.940531 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:15:00.940503    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac9525b_5d5a_4318_89c9_6cb675f8ad81.slice/crio-772f466ea5760d7088fdab4671d8727662c7edb0ac34902df185ccf75d41f432 WatchSource:0}: Error finding container 772f466ea5760d7088fdab4671d8727662c7edb0ac34902df185ccf75d41f432: Status 404 returned error can't find the container with id 772f466ea5760d7088fdab4671d8727662c7edb0ac34902df185ccf75d41f432
Nov 03 04:15:01.025269 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.025238    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"]
Nov 03 04:15:01.026086 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:15:01.026060    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4774c8e_c4c5_4229_8396_9682397df7b7.slice/crio-acfcd7fadf534e7708b14803c805f01b897e162993fdfafe6fcc43d40f2da2b6 WatchSource:0}: Error finding container acfcd7fadf534e7708b14803c805f01b897e162993fdfafe6fcc43d40f2da2b6: Status 404 returned error can't find the container with id acfcd7fadf534e7708b14803c805f01b897e162993fdfafe6fcc43d40f2da2b6
Nov 03 04:15:01.049192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.049163    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"]
Nov 03 04:15:01.052456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.052431    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"]
Nov 03 04:15:01.064855 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:15:01.064779    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bd31a8_8e83_4da8_9da8_bcac2d804f0c.slice/crio-683858cb912ac9e1f04ea3082b66818bfa59153e0be3c748c9305ba948cc2ec7 WatchSource:0}: Error finding container 683858cb912ac9e1f04ea3082b66818bfa59153e0be3c748c9305ba948cc2ec7: Status 404 returned error can't find the container with id 683858cb912ac9e1f04ea3082b66818bfa59153e0be3c748c9305ba948cc2ec7
Nov 03 04:15:01.065521 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:15:01.065408    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206918be_ffb7_4338_bd7f_f4950591052e.slice/crio-7c2339b83d3d44ad54478f0f05bc63eccedb674064bd66e53e1086f164aa4069 WatchSource:0}: Error finding container 7c2339b83d3d44ad54478f0f05bc63eccedb674064bd66e53e1086f164aa4069: Status 404 returned error can't find the container with id 7c2339b83d3d44ad54478f0f05bc63eccedb674064bd66e53e1086f164aa4069
Nov 03 04:15:01.224534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.223833    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"]
Nov 03 04:15:01.233708 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.233679    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"]
Nov 03 04:15:01.234165 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:15:01.234089    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84972876_2a60_4466_b92d_0a21a7237bd2.slice/crio-cd8ec4ea31a1eccbbe56102078711a2dafd2b34c1fdfec8d756bdc0a4fa3e385 WatchSource:0}: Error finding container cd8ec4ea31a1eccbbe56102078711a2dafd2b34c1fdfec8d756bdc0a4fa3e385: Status 404 returned error can't find the container with id cd8ec4ea31a1eccbbe56102078711a2dafd2b34c1fdfec8d756bdc0a4fa3e385
Nov 03 04:15:01.439785 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.439750    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:01.807335 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.807295    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerStarted","Data":"ae07a608adb46fe6ef47e3bda1eacdb528474fefb14ba7ae270f34baa2032afb"}
Nov 03 04:15:01.808032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.807356    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerStarted","Data":"cd8ec4ea31a1eccbbe56102078711a2dafd2b34c1fdfec8d756bdc0a4fa3e385"}
Nov 03 04:15:01.809169 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.809144    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerStarted","Data":"7cb4f02b60c22462bd3e84a7ad6ec93c0737d8c6218e12eb5d7fe43868b33bca"}
Nov 03 04:15:01.809290 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.809194    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerStarted","Data":"ea831a32c3eadeec9e279b528dc21c442095ba9531925d5049b13f431c321305"}
Nov 03 04:15:01.810665 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.810643    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerStarted","Data":"f6cd31a619d2ac20f50836c8b7771de7b71d5fd152f00c0789ea66c5b73da0dc"}
Nov 03 04:15:01.810767 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.810680    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerStarted","Data":"683858cb912ac9e1f04ea3082b66818bfa59153e0be3c748c9305ba948cc2ec7"}
Nov 03 04:15:01.812243 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.812222    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerStarted","Data":"fd9d5255aae21df6c42ebfc5c88a3c8d9a6c690c12a4ebc34ff9c71530cdcd83"}
Nov 03 04:15:01.812348 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.812260    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerStarted","Data":"7c2339b83d3d44ad54478f0f05bc63eccedb674064bd66e53e1086f164aa4069"}
Nov 03 04:15:01.813625 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.813607    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerStarted","Data":"801f0b32c87a3285d8027dcdffa44a94c3312eb8c46c9f818e3cfc92641ed84a"}
Nov 03 04:15:01.813718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.813664    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerStarted","Data":"acfcd7fadf534e7708b14803c805f01b897e162993fdfafe6fcc43d40f2da2b6"}
Nov 03 04:15:01.815025 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.815004    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerStarted","Data":"25b91c3ca3c4ab9297a011462d190eab2add81b1a87200e98d95cc76e9211822"}
Nov 03 04:15:01.815115 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.815043    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerStarted","Data":"772f466ea5760d7088fdab4671d8727662c7edb0ac34902df185ccf75d41f432"}
Nov 03 04:15:01.927621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:01.927567    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" podStartSLOduration=1.9275383339999999 podStartE2EDuration="1.927538334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:01.925017318 +0000 UTC m=+4393.155909783" watchObservedRunningTime="2026-11-03 04:15:01.927538334 +0000 UTC m=+4393.158430752"
Nov 03 04:15:02.044333 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.044236    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" podStartSLOduration=2.04422217 podStartE2EDuration="2.04422217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.041763972 +0000 UTC m=+4393.272656391" watchObservedRunningTime="2026-11-03 04:15:02.04422217 +0000 UTC m=+4393.275114588"
Nov 03 04:15:02.160662 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.160611    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" podStartSLOduration=2.160598001 podStartE2EDuration="2.160598001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.158617834 +0000 UTC m=+4393.389510252" watchObservedRunningTime="2026-11-03 04:15:02.160598001 +0000 UTC m=+4393.391490418"
Nov 03 04:15:02.277666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.277614    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" podStartSLOduration=2.277598729 podStartE2EDuration="2.277598729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.275028098 +0000 UTC m=+4393.505920517" watchObservedRunningTime="2026-11-03 04:15:02.277598729 +0000 UTC m=+4393.508491147"
Nov 03 04:15:02.393700 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.393650    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" podStartSLOduration=2.393634402 podStartE2EDuration="2.393634402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.391901031 +0000 UTC m=+4393.622793450" watchObservedRunningTime="2026-11-03 04:15:02.393634402 +0000 UTC m=+4393.624526860"
Nov 03 04:15:02.511069 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.511017    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" podStartSLOduration=2.5110021270000003 podStartE2EDuration="2.511002127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:15:02.510681746 +0000 UTC m=+4393.741574166" watchObservedRunningTime="2026-11-03 04:15:02.511002127 +0000 UTC m=+4393.741894545"
Nov 03 04:15:02.780805 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.780769    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:02.823840 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.823807    5515 generic.go:356] "Generic (PLEG): container finished" podID="84972876-2a60-4466-b92d-0a21a7237bd2" containerID="ae07a608adb46fe6ef47e3bda1eacdb528474fefb14ba7ae270f34baa2032afb" exitCode=60
Nov 03 04:15:02.824082 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.823857    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerDied","Data":"ae07a608adb46fe6ef47e3bda1eacdb528474fefb14ba7ae270f34baa2032afb"}
Nov 03 04:15:02.824258 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.824228    5515 scope.go:122] "RemoveContainer" containerID="ae07a608adb46fe6ef47e3bda1eacdb528474fefb14ba7ae270f34baa2032afb"
Nov 03 04:15:02.825685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.825662    5515 generic.go:356] "Generic (PLEG): container finished" podID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerID="7cb4f02b60c22462bd3e84a7ad6ec93c0737d8c6218e12eb5d7fe43868b33bca" exitCode=60
Nov 03 04:15:02.825756 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.825716    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerDied","Data":"7cb4f02b60c22462bd3e84a7ad6ec93c0737d8c6218e12eb5d7fe43868b33bca"}
Nov 03 04:15:02.826010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.825993    5515 scope.go:122] "RemoveContainer" containerID="7cb4f02b60c22462bd3e84a7ad6ec93c0737d8c6218e12eb5d7fe43868b33bca"
Nov 03 04:15:02.827509 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.827476    5515 generic.go:356] "Generic (PLEG): container finished" podID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerID="f6cd31a619d2ac20f50836c8b7771de7b71d5fd152f00c0789ea66c5b73da0dc" exitCode=60
Nov 03 04:15:02.827639 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.827585    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerDied","Data":"f6cd31a619d2ac20f50836c8b7771de7b71d5fd152f00c0789ea66c5b73da0dc"}
Nov 03 04:15:02.827962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.827782    5515 scope.go:122] "RemoveContainer" containerID="f6cd31a619d2ac20f50836c8b7771de7b71d5fd152f00c0789ea66c5b73da0dc"
Nov 03 04:15:02.829864 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.829845    5515 generic.go:356] "Generic (PLEG): container finished" podID="206918be-ffb7-4338-bd7f-f4950591052e" containerID="fd9d5255aae21df6c42ebfc5c88a3c8d9a6c690c12a4ebc34ff9c71530cdcd83" exitCode=60
Nov 03 04:15:02.829950 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.829931    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerDied","Data":"fd9d5255aae21df6c42ebfc5c88a3c8d9a6c690c12a4ebc34ff9c71530cdcd83"}
Nov 03 04:15:02.830128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.830115    5515 scope.go:122] "RemoveContainer" containerID="fd9d5255aae21df6c42ebfc5c88a3c8d9a6c690c12a4ebc34ff9c71530cdcd83"
Nov 03 04:15:02.831533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.831515    5515 generic.go:356] "Generic (PLEG): container finished" podID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerID="801f0b32c87a3285d8027dcdffa44a94c3312eb8c46c9f818e3cfc92641ed84a" exitCode=60
Nov 03 04:15:02.831650 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.831597    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerDied","Data":"801f0b32c87a3285d8027dcdffa44a94c3312eb8c46c9f818e3cfc92641ed84a"}
Nov 03 04:15:02.831772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.831759    5515 scope.go:122] "RemoveContainer" containerID="801f0b32c87a3285d8027dcdffa44a94c3312eb8c46c9f818e3cfc92641ed84a"
Nov 03 04:15:02.834850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.833674    5515 generic.go:356] "Generic (PLEG): container finished" podID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerID="25b91c3ca3c4ab9297a011462d190eab2add81b1a87200e98d95cc76e9211822" exitCode=60
Nov 03 04:15:02.834850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.833764    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerDied","Data":"25b91c3ca3c4ab9297a011462d190eab2add81b1a87200e98d95cc76e9211822"}
Nov 03 04:15:02.834850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:02.834040    5515 scope.go:122] "RemoveContainer" containerID="25b91c3ca3c4ab9297a011462d190eab2add81b1a87200e98d95cc76e9211822"
Nov 03 04:15:03.670950 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.670877    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:03.814191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.814128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:03.844776 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.844735    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerStarted","Data":"e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08"}
Nov 03 04:15:03.846883 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.846859    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerStarted","Data":"80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4"}
Nov 03 04:15:03.848809 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.848784    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerStarted","Data":"f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7"}
Nov 03 04:15:03.850664 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.850646    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerStarted","Data":"5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef"}
Nov 03 04:15:03.852640 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.852620    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerStarted","Data":"b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c"}
Nov 03 04:15:03.854397 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:03.854379    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerStarted","Data":"ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21"}
Nov 03 04:15:04.863333 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.863300    5515 generic.go:356] "Generic (PLEG): container finished" podID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerID="80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4" exitCode=60
Nov 03 04:15:04.863859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.863370    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerDied","Data":"80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4"}
Nov 03 04:15:04.863859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.863420    5515 scope.go:122] "RemoveContainer" containerID="7cb4f02b60c22462bd3e84a7ad6ec93c0737d8c6218e12eb5d7fe43868b33bca"
Nov 03 04:15:04.863859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.863768    5515 scope.go:122] "RemoveContainer" containerID="80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4"
Nov 03 04:15:04.864156 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:04.864132    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7_odh-ai-gateway-infra(86fedb24-5cd5-4edc-befe-376f05ea6231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231"
Nov 03 04:15:04.865376 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.865352    5515 generic.go:356] "Generic (PLEG): container finished" podID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerID="f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7" exitCode=60
Nov 03 04:15:04.865456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.865428    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerDied","Data":"f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7"}
Nov 03 04:15:04.865683 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.865669    5515 scope.go:122] "RemoveContainer" containerID="f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7"
Nov 03 04:15:04.865945 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:04.865926    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6_odh-ai-gateway-infra(e4bd31a8-8e83-4da8-9da8-bcac2d804f0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c"
Nov 03 04:15:04.867294 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.867272    5515 generic.go:356] "Generic (PLEG): container finished" podID="206918be-ffb7-4338-bd7f-f4950591052e" containerID="5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef" exitCode=60
Nov 03 04:15:04.867390 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.867351    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerDied","Data":"5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef"}
Nov 03 04:15:04.867531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.867518    5515 scope.go:122] "RemoveContainer" containerID="5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef"
Nov 03 04:15:04.867821 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:04.867795    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt_odh-ai-gateway-infra(206918be-ffb7-4338-bd7f-f4950591052e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" podUID="206918be-ffb7-4338-bd7f-f4950591052e"
Nov 03 04:15:04.869532 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.869512    5515 generic.go:356] "Generic (PLEG): container finished" podID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerID="b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c" exitCode=60
Nov 03 04:15:04.869645 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.869592    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerDied","Data":"b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c"}
Nov 03 04:15:04.869781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.869766    5515 scope.go:122] "RemoveContainer" containerID="b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c"
Nov 03 04:15:04.870009 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:04.869985    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj_odh-ai-gateway-infra(f4774c8e-c4c5-4229-8396-9682397df7b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7"
Nov 03 04:15:04.871637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.871618    5515 generic.go:356] "Generic (PLEG): container finished" podID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerID="ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21" exitCode=60
Nov 03 04:15:04.871741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.871701    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerDied","Data":"ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21"}
Nov 03 04:15:04.871924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.871909    5515 scope.go:122] "RemoveContainer" containerID="ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21"
Nov 03 04:15:04.872204 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:04.872178    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm_odh-ai-gateway-infra(cac9525b-5d5a-4318-89c9-6cb675f8ad81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81"
Nov 03 04:15:04.874257 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.874226    5515 generic.go:356] "Generic (PLEG): container finished" podID="84972876-2a60-4466-b92d-0a21a7237bd2" containerID="e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08" exitCode=60
Nov 03 04:15:04.874355 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.874310    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerDied","Data":"e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08"}
Nov 03 04:15:04.874497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.874483    5515 scope.go:122] "RemoveContainer" containerID="e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08"
Nov 03 04:15:04.874713 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:04.874694    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p_odh-ai-gateway-infra(84972876-2a60-4466-b92d-0a21a7237bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" podUID="84972876-2a60-4466-b92d-0a21a7237bd2"
Nov 03 04:15:04.877696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.877679    5515 scope.go:122] "RemoveContainer" containerID="f6cd31a619d2ac20f50836c8b7771de7b71d5fd152f00c0789ea66c5b73da0dc"
Nov 03 04:15:04.890469 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.890455    5515 scope.go:122] "RemoveContainer" containerID="fd9d5255aae21df6c42ebfc5c88a3c8d9a6c690c12a4ebc34ff9c71530cdcd83"
Nov 03 04:15:04.903168 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.903148    5515 scope.go:122] "RemoveContainer" containerID="801f0b32c87a3285d8027dcdffa44a94c3312eb8c46c9f818e3cfc92641ed84a"
Nov 03 04:15:04.915935 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.915911    5515 scope.go:122] "RemoveContainer" containerID="25b91c3ca3c4ab9297a011462d190eab2add81b1a87200e98d95cc76e9211822"
Nov 03 04:15:04.928923 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:04.928900    5515 scope.go:122] "RemoveContainer" containerID="ae07a608adb46fe6ef47e3bda1eacdb528474fefb14ba7ae270f34baa2032afb"
Nov 03 04:15:05.886914 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:05.886888    5515 scope.go:122] "RemoveContainer" containerID="80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4"
Nov 03 04:15:05.887398 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:05.887244    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7_odh-ai-gateway-infra(86fedb24-5cd5-4edc-befe-376f05ea6231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231"
Nov 03 04:15:05.888602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:05.888576    5515 scope.go:122] "RemoveContainer" containerID="f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7"
Nov 03 04:15:05.888875 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:05.888854    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6_odh-ai-gateway-infra(e4bd31a8-8e83-4da8-9da8-bcac2d804f0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c"
Nov 03 04:15:05.890229 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:05.890208    5515 scope.go:122] "RemoveContainer" containerID="5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef"
Nov 03 04:15:05.890496 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:05.890471    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt_odh-ai-gateway-infra(206918be-ffb7-4338-bd7f-f4950591052e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" podUID="206918be-ffb7-4338-bd7f-f4950591052e"
Nov 03 04:15:05.891739 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:05.891718    5515 scope.go:122] "RemoveContainer" containerID="b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c"
Nov 03 04:15:05.891984 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:05.891963    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj_odh-ai-gateway-infra(f4774c8e-c4c5-4229-8396-9682397df7b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7"
Nov 03 04:15:05.893141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:05.893124    5515 scope.go:122] "RemoveContainer" containerID="ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21"
Nov 03 04:15:05.893382 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:05.893357    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm_odh-ai-gateway-infra(cac9525b-5d5a-4318-89c9-6cb675f8ad81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81"
Nov 03 04:15:05.894643 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:05.894627    5515 scope.go:122] "RemoveContainer" containerID="e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08"
Nov 03 04:15:05.894841 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:05.894824    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p_odh-ai-gateway-infra(84972876-2a60-4466-b92d-0a21a7237bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" podUID="84972876-2a60-4466-b92d-0a21a7237bd2"
Nov 03 04:15:06.339685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:06.339602    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:11.125032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:11.125000    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:15:11.164388 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:11.164351    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:13.223687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:13.223647    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:13.367274 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:13.367244    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:14.887160 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:14.887131    5515 scope.go:122] "RemoveContainer" containerID="e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08"
Nov 03 04:15:15.886562 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:15.886524    5515 scope.go:122] "RemoveContainer" containerID="ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21"
Nov 03 04:15:15.961428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:15.961394    5515 generic.go:356] "Generic (PLEG): container finished" podID="84972876-2a60-4466-b92d-0a21a7237bd2" containerID="d0ee83ee8891e501b785a959e74078a5a170de9330371fc8cc8365c4992b9d4e" exitCode=60
Nov 03 04:15:15.961911 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:15.961438    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerDied","Data":"d0ee83ee8891e501b785a959e74078a5a170de9330371fc8cc8365c4992b9d4e"}
Nov 03 04:15:15.961911 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:15.961523    5515 scope.go:122] "RemoveContainer" containerID="e33c6c193cb684a962fcb5eafab67fe0f8720199245a61d40c1a64da0e5cbe08"
Nov 03 04:15:15.962037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:15.961917    5515 scope.go:122] "RemoveContainer" containerID="d0ee83ee8891e501b785a959e74078a5a170de9330371fc8cc8365c4992b9d4e"
Nov 03 04:15:15.962285 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:15.962266    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p_odh-ai-gateway-infra(84972876-2a60-4466-b92d-0a21a7237bd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" podUID="84972876-2a60-4466-b92d-0a21a7237bd2"
Nov 03 04:15:16.070569 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:16.070472    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:16.887506 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:16.887482    5515 scope.go:122] "RemoveContainer" containerID="b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c"
Nov 03 04:15:16.975300 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:16.975271    5515 generic.go:356] "Generic (PLEG): container finished" podID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerID="a8e26bcf85b6cb4201f8d15ffaca957b4fd69d91270f8ab5c9391fc9f6b52e90" exitCode=60
Nov 03 04:15:16.975757 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:16.975347    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerDied","Data":"a8e26bcf85b6cb4201f8d15ffaca957b4fd69d91270f8ab5c9391fc9f6b52e90"}
Nov 03 04:15:16.975757 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:16.975411    5515 scope.go:122] "RemoveContainer" containerID="ba65643c622e8ca348168757b970c7dea47f0a54ee287bc54fa8180112db0d21"
Nov 03 04:15:16.975879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:16.975860    5515 scope.go:122] "RemoveContainer" containerID="a8e26bcf85b6cb4201f8d15ffaca957b4fd69d91270f8ab5c9391fc9f6b52e90"
Nov 03 04:15:16.976218 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:16.976196    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm_odh-ai-gateway-infra(cac9525b-5d5a-4318-89c9-6cb675f8ad81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81"
Nov 03 04:15:17.090556 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.090518    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"]
Nov 03 04:15:17.248349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.248322    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:17.305422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.305389    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr\" (UniqueName: \"kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr\") pod \"84972876-2a60-4466-b92d-0a21a7237bd2\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") "
Nov 03 04:15:17.305606 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.305592    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca\") pod \"84972876-2a60-4466-b92d-0a21a7237bd2\" (UID: \"84972876-2a60-4466-b92d-0a21a7237bd2\") "
Nov 03 04:15:17.305939 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.305910    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca" pod "84972876-2a60-4466-b92d-0a21a7237bd2" (UID: "84972876-2a60-4466-b92d-0a21a7237bd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:15:17.306054 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.306008    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84972876-2a60-4466-b92d-0a21a7237bd2-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:17.307661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.307633    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr" pod "84972876-2a60-4466-b92d-0a21a7237bd2" (UID: "84972876-2a60-4466-b92d-0a21a7237bd2"). InnerVolumeSpecName "kube-api-access-sj8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:15:17.407878 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.407786    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8sr\" (UniqueName: \"kubernetes.io/projected/84972876-2a60-4466-b92d-0a21a7237bd2-kube-api-access-sj8sr\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:17.887607 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.887580    5515 scope.go:122] "RemoveContainer" containerID="80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4"
Nov 03 04:15:17.887764 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.887635    5515 scope.go:122] "RemoveContainer" containerID="f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7"
Nov 03 04:15:17.887807 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.887758    5515 scope.go:122] "RemoveContainer" containerID="5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef"
Nov 03 04:15:17.983264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.983236    5515 generic.go:356] "Generic (PLEG): container finished" podID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerID="83188abf59e69be9723d9396400d91d719985cf84a840ce2800aab21f0646a99" exitCode=60
Nov 03 04:15:17.983642 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.983309    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerDied","Data":"83188abf59e69be9723d9396400d91d719985cf84a840ce2800aab21f0646a99"}
Nov 03 04:15:17.983642 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.983404    5515 scope.go:122] "RemoveContainer" containerID="b666f4ffba7895890e2278d173ab5ddb645aedf6e595db6c19b869f34d23599c"
Nov 03 04:15:17.983772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.983750    5515 scope.go:122] "RemoveContainer" containerID="83188abf59e69be9723d9396400d91d719985cf84a840ce2800aab21f0646a99"
Nov 03 04:15:17.984066 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:17.984040    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj_odh-ai-gateway-infra(f4774c8e-c4c5-4229-8396-9682397df7b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7"
Nov 03 04:15:17.986306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.986283    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"
Nov 03 04:15:17.986396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.986303    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p" event={"ID":"84972876-2a60-4466-b92d-0a21a7237bd2","Type":"ContainerDied","Data":"cd8ec4ea31a1eccbbe56102078711a2dafd2b34c1fdfec8d756bdc0a4fa3e385"}
Nov 03 04:15:17.997953 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:17.997936    5515 scope.go:122] "RemoveContainer" containerID="d0ee83ee8891e501b785a959e74078a5a170de9330371fc8cc8365c4992b9d4e"
Nov 03 04:15:18.111420 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.111389    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"]
Nov 03 04:15:18.214930 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.214872    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"]
Nov 03 04:15:18.219544 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.219516    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-efa9bf-29894655-vnw7p"]
Nov 03 04:15:18.377138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.377110    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:18.419834 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.419802    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77\" (UniqueName: \"kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77\") pod \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") "
Nov 03 04:15:18.420046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.419861    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca\") pod \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\" (UID: \"cac9525b-5d5a-4318-89c9-6cb675f8ad81\") "
Nov 03 04:15:18.420585 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.420525    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca" pod "cac9525b-5d5a-4318-89c9-6cb675f8ad81" (UID: "cac9525b-5d5a-4318-89c9-6cb675f8ad81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:15:18.422377 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.422350    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77" pod "cac9525b-5d5a-4318-89c9-6cb675f8ad81" (UID: "cac9525b-5d5a-4318-89c9-6cb675f8ad81"). InnerVolumeSpecName "kube-api-access-d7k77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:15:18.521513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.521480    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k77\" (UniqueName: \"kubernetes.io/projected/cac9525b-5d5a-4318-89c9-6cb675f8ad81-kube-api-access-d7k77\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:18.521513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.521511    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac9525b-5d5a-4318-89c9-6cb675f8ad81-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:18.895441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.895412    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" path="/var/lib/kubelet/pods/84972876-2a60-4466-b92d-0a21a7237bd2/volumes"
Nov 03 04:15:18.994696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.994664    5515 generic.go:356] "Generic (PLEG): container finished" podID="206918be-ffb7-4338-bd7f-f4950591052e" containerID="7494d753db48e7135762a36b30789fd4d706a13ffcc139ca675f5bdd040b83e5" exitCode=60
Nov 03 04:15:18.995172 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.994738    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerDied","Data":"7494d753db48e7135762a36b30789fd4d706a13ffcc139ca675f5bdd040b83e5"}
Nov 03 04:15:18.995172 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.994802    5515 scope.go:122] "RemoveContainer" containerID="5f118ed687a6a3b114a01f6a712957664a141b565a923cf2b899a344133a71ef"
Nov 03 04:15:18.995293 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.995173    5515 scope.go:122] "RemoveContainer" containerID="7494d753db48e7135762a36b30789fd4d706a13ffcc139ca675f5bdd040b83e5"
Nov 03 04:15:18.995605 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:18.995582    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt_odh-ai-gateway-infra(206918be-ffb7-4338-bd7f-f4950591052e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" podUID="206918be-ffb7-4338-bd7f-f4950591052e"
Nov 03 04:15:18.997886 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.997865    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"
Nov 03 04:15:18.997990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:18.997895    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm" event={"ID":"cac9525b-5d5a-4318-89c9-6cb675f8ad81","Type":"ContainerDied","Data":"772f466ea5760d7088fdab4671d8727662c7edb0ac34902df185ccf75d41f432"}
Nov 03 04:15:19.001170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.001145    5515 generic.go:356] "Generic (PLEG): container finished" podID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerID="834006c87de6fc3b814102ff7fa3bcbeadc298de6b63eadd3186c818f977b183" exitCode=60
Nov 03 04:15:19.001285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.001214    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerDied","Data":"834006c87de6fc3b814102ff7fa3bcbeadc298de6b63eadd3186c818f977b183"}
Nov 03 04:15:19.001663 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.001645    5515 scope.go:122] "RemoveContainer" containerID="834006c87de6fc3b814102ff7fa3bcbeadc298de6b63eadd3186c818f977b183"
Nov 03 04:15:19.001950 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:19.001928    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7_odh-ai-gateway-infra(86fedb24-5cd5-4edc-befe-376f05ea6231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231"
Nov 03 04:15:19.003292 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.003272    5515 generic.go:356] "Generic (PLEG): container finished" podID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerID="0583f9a3e0a3163c68a3e261de90571970d7238eb9d05b703fc29f5bdf662f25" exitCode=60
Nov 03 04:15:19.003381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.003303    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerDied","Data":"0583f9a3e0a3163c68a3e261de90571970d7238eb9d05b703fc29f5bdf662f25"}
Nov 03 04:15:19.003657 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.003643    5515 scope.go:122] "RemoveContainer" containerID="0583f9a3e0a3163c68a3e261de90571970d7238eb9d05b703fc29f5bdf662f25"
Nov 03 04:15:19.003931 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:19.003902    5515 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6_odh-ai-gateway-infra(e4bd31a8-8e83-4da8-9da8-bcac2d804f0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c"
Nov 03 04:15:19.012427 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.012406    5515 scope.go:122] "RemoveContainer" containerID="a8e26bcf85b6cb4201f8d15ffaca957b4fd69d91270f8ab5c9391fc9f6b52e90"
Nov 03 04:15:19.022169 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.022150    5515 scope.go:122] "RemoveContainer" containerID="80b9d74de820837bc4a9d2717452adcf1e4d5b42f58e42fdb878650e0a1665c4"
Nov 03 04:15:19.035532 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.035512    5515 scope.go:122] "RemoveContainer" containerID="f19d4a216c4978f0a618b08d25b97556f9bc2c0f6d9ec558d94b508c696f97a7"
Nov 03 04:15:19.110351 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.110316    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"]
Nov 03 04:15:19.274320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.274248    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:19.333327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.333287    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca\") pod \"f4774c8e-c4c5-4229-8396-9682397df7b7\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") "
Nov 03 04:15:19.333528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.333411    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl\" (UniqueName: \"kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl\") pod \"f4774c8e-c4c5-4229-8396-9682397df7b7\" (UID: \"f4774c8e-c4c5-4229-8396-9682397df7b7\") "
Nov 03 04:15:19.333767 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.333745    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca" pod "f4774c8e-c4c5-4229-8396-9682397df7b7" (UID: "f4774c8e-c4c5-4229-8396-9682397df7b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:15:19.333952 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.333938    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4774c8e-c4c5-4229-8396-9682397df7b7-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:19.335753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.335713    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl" pod "f4774c8e-c4c5-4229-8396-9682397df7b7" (UID: "f4774c8e-c4c5-4229-8396-9682397df7b7"). InnerVolumeSpecName "kube-api-access-c4qnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:15:19.435487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.435449    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qnl\" (UniqueName: \"kubernetes.io/projected/f4774c8e-c4c5-4229-8396-9682397df7b7-kube-api-access-c4qnl\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:19.580484 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.580402    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"]
Nov 03 04:15:19.588852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:19.588825    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-de27f5-29894655-jmszm"]
Nov 03 04:15:20.017721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.017682    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"
Nov 03 04:15:20.018205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.017678    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj" event={"ID":"f4774c8e-c4c5-4229-8396-9682397df7b7","Type":"ContainerDied","Data":"acfcd7fadf534e7708b14803c805f01b897e162993fdfafe6fcc43d40f2da2b6"}
Nov 03 04:15:20.018205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.017869    5515 scope.go:122] "RemoveContainer" containerID="83188abf59e69be9723d9396400d91d719985cf84a840ce2800aab21f0646a99"
Nov 03 04:15:20.125032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.124983    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"]
Nov 03 04:15:20.193038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.193001    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"]
Nov 03 04:15:20.199071 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.199041    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7ed2d6-29894655-xm2lj"]
Nov 03 04:15:20.242781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.242749    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"]
Nov 03 04:15:20.340583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.339965    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:20.342752 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.342724    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:20.467380 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.464450    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46\" (UniqueName: \"kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46\") pod \"206918be-ffb7-4338-bd7f-f4950591052e\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") "
Nov 03 04:15:20.467380 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.465147    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca\") pod \"206918be-ffb7-4338-bd7f-f4950591052e\" (UID: \"206918be-ffb7-4338-bd7f-f4950591052e\") "
Nov 03 04:15:20.467380 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.465490    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca" pod "206918be-ffb7-4338-bd7f-f4950591052e" (UID: "206918be-ffb7-4338-bd7f-f4950591052e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:15:20.467380 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.466292    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206918be-ffb7-4338-bd7f-f4950591052e-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:20.480337 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.477931    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46" pod "206918be-ffb7-4338-bd7f-f4950591052e" (UID: "206918be-ffb7-4338-bd7f-f4950591052e"). InnerVolumeSpecName "kube-api-access-9nb46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:15:20.512599 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.505698    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"]
Nov 03 04:15:20.556399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.556327    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:20.568160 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.568129    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nb46\" (UniqueName: \"kubernetes.io/projected/206918be-ffb7-4338-bd7f-f4950591052e-kube-api-access-9nb46\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:20.717440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.717399    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca\") pod \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") "
Nov 03 04:15:20.732971 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.717633    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk\" (UniqueName: \"kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk\") pod \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\" (UID: \"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c\") "
Nov 03 04:15:20.732971 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.720886    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk" pod "e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" (UID: "e4bd31a8-8e83-4da8-9da8-bcac2d804f0c"). InnerVolumeSpecName "kube-api-access-4czhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:15:20.736840 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.736806    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca" pod "e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" (UID: "e4bd31a8-8e83-4da8-9da8-bcac2d804f0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:15:20.772341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.772312    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:20.825629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.824781    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czhk\" (UniqueName: \"kubernetes.io/projected/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-kube-api-access-4czhk\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:20.825629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.824827    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:20.899229 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.899194    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" path="/var/lib/kubelet/pods/cac9525b-5d5a-4318-89c9-6cb675f8ad81/volumes"
Nov 03 04:15:20.899677 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.899662    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" path="/var/lib/kubelet/pods/f4774c8e-c4c5-4229-8396-9682397df7b7/volumes"
Nov 03 04:15:20.925685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.925659    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h\" (UniqueName: \"kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h\") pod \"86fedb24-5cd5-4edc-befe-376f05ea6231\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") "
Nov 03 04:15:20.926062 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.926037    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca\") pod \"86fedb24-5cd5-4edc-befe-376f05ea6231\" (UID: \"86fedb24-5cd5-4edc-befe-376f05ea6231\") "
Nov 03 04:15:20.926441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.926344    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca" pod "86fedb24-5cd5-4edc-befe-376f05ea6231" (UID: "86fedb24-5cd5-4edc-befe-376f05ea6231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:15:20.926578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.926532    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fedb24-5cd5-4edc-befe-376f05ea6231-maas-api-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:20.928096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:20.928071    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h" pod "86fedb24-5cd5-4edc-befe-376f05ea6231" (UID: "86fedb24-5cd5-4edc-befe-376f05ea6231"). InnerVolumeSpecName "kube-api-access-q722h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:15:20.991223 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:20.991177    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:15:21.027956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.027922    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q722h\" (UniqueName: \"kubernetes.io/projected/86fedb24-5cd5-4edc-befe-376f05ea6231-kube-api-access-q722h\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:15:21.036697 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.036658    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7" event={"ID":"86fedb24-5cd5-4edc-befe-376f05ea6231","Type":"ContainerDied","Data":"ea831a32c3eadeec9e279b528dc21c442095ba9531925d5049b13f431c321305"}
Nov 03 04:15:21.036863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.036668    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"
Nov 03 04:15:21.036863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.036721    5515 scope.go:122] "RemoveContainer" containerID="834006c87de6fc3b814102ff7fa3bcbeadc298de6b63eadd3186c818f977b183"
Nov 03 04:15:21.038203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.038178    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"
Nov 03 04:15:21.038327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.038205    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6" event={"ID":"e4bd31a8-8e83-4da8-9da8-bcac2d804f0c","Type":"ContainerDied","Data":"683858cb912ac9e1f04ea3082b66818bfa59153e0be3c748c9305ba948cc2ec7"}
Nov 03 04:15:21.039878 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.039858    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"
Nov 03 04:15:21.040024 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.039880    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt" event={"ID":"206918be-ffb7-4338-bd7f-f4950591052e","Type":"ContainerDied","Data":"7c2339b83d3d44ad54478f0f05bc63eccedb674064bd66e53e1086f164aa4069"}
Nov 03 04:15:21.048081 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.048063    5515 scope.go:122] "RemoveContainer" containerID="0583f9a3e0a3163c68a3e261de90571970d7238eb9d05b703fc29f5bdf662f25"
Nov 03 04:15:21.056872 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.056849    5515 scope.go:122] "RemoveContainer" containerID="7494d753db48e7135762a36b30789fd4d706a13ffcc139ca675f5bdd040b83e5"
Nov 03 04:15:21.240577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.240245    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"]
Nov 03 04:15:21.248524 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.248489    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5a6246-29894655-nkhm7"]
Nov 03 04:15:21.341867 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.341813    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"]
Nov 03 04:15:21.350415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.350381    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8540f6-29894655-jjnzt"]
Nov 03 04:15:21.462348 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.462311    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"]
Nov 03 04:15:21.465819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:21.465791    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-255c49-29894655-ffwc6"]
Nov 03 04:15:22.122715 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:22.122680    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:22.898089 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:22.898053    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206918be-ffb7-4338-bd7f-f4950591052e" path="/var/lib/kubelet/pods/206918be-ffb7-4338-bd7f-f4950591052e/volumes"
Nov 03 04:15:22.898524 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:22.898510    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" path="/var/lib/kubelet/pods/86fedb24-5cd5-4edc-befe-376f05ea6231/volumes"
Nov 03 04:15:22.899000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:22.898986    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" path="/var/lib/kubelet/pods/e4bd31a8-8e83-4da8-9da8-bcac2d804f0c/volumes"
Nov 03 04:15:26.955084 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:26.955053    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:29.988819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:29.988786    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:30.133851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:30.133818    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:31.574579 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:31.574528    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:31.734347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:31.734278    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:31.880704 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:31.880663    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:32.025756 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:32.025719    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:32.185816 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:32.185698    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:32.614536 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:32.614499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:33.123181 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:33.123144    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:35.617045 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:35.617013    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:37.232691 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:37.232650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:39.448258 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:39.448219    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:40.694017 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:40.693964    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:40.840047 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:40.840016    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:42.395158 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:42.395124    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:45.780993 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:45.780956    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:47.415537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:47.415478    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:49.346883 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:49.346851    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:49.930737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:49.930706    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:50.601873 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:50.601824    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:50.763506 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:50.763467    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:50.984034 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:15:50.983992    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:15:52.648916 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:52.648876    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:54.843515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:54.843481    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:55.021858 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:55.021826    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:55.174046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:55.174005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:57.874645 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:57.874613    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:58.087413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:58.087378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:58.272670 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:58.272378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:15:59.061583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:15:59.061529    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:03.055468 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:03.055431    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:03.751357 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:03.751322    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:16:05.447626 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:05.447587    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:08.254620 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:08.254583    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:12.987303 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:12.987263    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:13.150395 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:13.150364    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:13.294632 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:13.294529    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:13.571202 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:13.571117    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:18.007999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:18.007959    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:20.984643 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:16:20.984602    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:16:22.862215 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:22.862171    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:25.455560 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:25.455506    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:25.601407 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:25.601378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:25.745531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:25.745424    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:26.986867 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:26.986832    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:27.249786 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:27.249712    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:27.737265 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:27.737233    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:32.622098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:32.622057    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:37.454850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:37.454813    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:40.027208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:40.027174    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:42.367990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:42.367948    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:46.491375 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:46.491344    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:47.201098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:47.201062    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:50.343422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:50.343393    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:16:50.346008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:50.345987    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:16:50.388693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:50.388669    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:16:50.390905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:50.390887    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:16:50.984755 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:16:50.984720    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:16:51.521718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:51.521682    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:53.230685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:53.230601    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:53.675183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:53.675149    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:16:57.076582 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:57.075974    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:57.233512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:57.233468    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:57.292407 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:57.292367    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:16:57.695574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:57.694633    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:16:58.306140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:58.306108    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:58.716848 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:58.716812    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:16:59.231727 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:59.231697    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"]
Nov 03 04:16:59.231936 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:59.231912    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy" containerID="cri-o://4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d" gracePeriod=30
Nov 03 04:16:59.241263 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:16:59.241237    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:02.626372 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:02.626341    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:03.787889 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:03.787858    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:04.512511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.512447    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:17:04.583143 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583116    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583310 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583220    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583310 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583289    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583310    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583359    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583422    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583655 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583467    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583655 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583511    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583655 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583633    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert\") pod \"e2c637e7-674e-4ff9-840e-586143a6a614\" (UID: \"e2c637e7-674e-4ff9-840e-586143a6a614\") "
Nov 03 04:17:04.583834 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583659    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:04.583834 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583682    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:04.584184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583875    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:04.584184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.583891    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:04.584184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.584168    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:17:04.584433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.584412    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.584512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.584439    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.584512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.584455    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.584512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.584470    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.584512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.584484    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c637e7-674e-4ff9-840e-586143a6a614-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.585635 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.585609    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:17:04.585841 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.585816    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:04.585945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.585923    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "kube-api-access-rxdx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:17:04.586165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.586148    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token" pod "e2c637e7-674e-4ff9-840e-586143a6a614" (UID: "e2c637e7-674e-4ff9-840e-586143a6a614"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:17:04.686479 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.686444    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.686479 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.686470    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c637e7-674e-4ff9-840e-586143a6a614-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.686479 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.686479    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c637e7-674e-4ff9-840e-586143a6a614-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.686764 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.686490    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdx5\" (UniqueName: \"kubernetes.io/projected/e2c637e7-674e-4ff9-840e-586143a6a614-kube-api-access-rxdx5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:04.804986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.804897    5515 generic.go:356] "Generic (PLEG): container finished" podID="e2c637e7-674e-4ff9-840e-586143a6a614" containerID="4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d" exitCode=0
Nov 03 04:17:04.804986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.804946    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"
Nov 03 04:17:04.805450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.804980    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" event={"ID":"e2c637e7-674e-4ff9-840e-586143a6a614","Type":"ContainerDied","Data":"4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d"}
Nov 03 04:17:04.805450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.805038    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq" event={"ID":"e2c637e7-674e-4ff9-840e-586143a6a614","Type":"ContainerDied","Data":"f3d981667b60fe5f2c810eddcac9e0a32f6423b07ff4ea9a8b5c87d189bbd0a0"}
Nov 03 04:17:04.805450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.805062    5515 scope.go:122] "RemoveContainer" containerID="4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d"
Nov 03 04:17:04.815755 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.815737    5515 scope.go:122] "RemoveContainer" containerID="4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d"
Nov 03 04:17:04.816067 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:17:04.816047    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d\": container with ID starting with 4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d not found: ID does not exist" containerID="4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d"
Nov 03 04:17:04.816115 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.816075    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d"} err="failed to get container status \"4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d\": rpc error: code = NotFound desc = could not find container \"4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d\": container with ID starting with 4b97a82a52c04a82cf3d012da4e52115d5b236e2020aebe062f77cb07385919d not found: ID does not exist"
Nov 03 04:17:04.975030 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.975006    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"]
Nov 03 04:17:04.977601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:04.977582    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8540f6-openshift-default-56d47784c9-fp8cq"]
Nov 03 04:17:06.897170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:06.897136    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" path="/var/lib/kubelet/pods/e2c637e7-674e-4ff9-840e-586143a6a614/volumes"
Nov 03 04:17:08.933378 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:08.933349    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:13.764306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:13.764264    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:13.931004 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:13.930972    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:18.370610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.370574    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"]
Nov 03 04:17:18.371152 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371138    5515 cpu_manager.go:414] "containerMap: removing container" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371154    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371169    5515 cpu_manager.go:414] "containerMap: removing container" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371174    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371182    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371188    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371196    5515 cpu_manager.go:414] "containerMap: removing container" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371200    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371206    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371211    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371220    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371225    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371232    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371237    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371243    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371247    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371258    5515 cpu_manager.go:414] "containerMap: removing container" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371263    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371277    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371281    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371290    5515 cpu_manager.go:414] "containerMap: removing container" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371294    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371300    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371305    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371314    5515 cpu_manager.go:414] "containerMap: removing container" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371318    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371326    5515 cpu_manager.go:414] "containerMap: removing container" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371330    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371344    5515 cpu_manager.go:414] "containerMap: removing container" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371348    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371356    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371360    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371439    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c637e7-674e-4ff9-840e-586143a6a614" containerName="istio-proxy"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371449    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371455    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371460    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371466    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371472    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371479    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371484    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371489    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371493    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371501    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371508    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371512    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:17:18.371535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.371517    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:17:18.376572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.376536    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.430470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.430444    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3c1cec-serving-cert\""
Nov 03 04:17:18.447639 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.447609    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.447808 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.447675    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtms\" (UniqueName: \"kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.447808 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.447784    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.447990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.447971    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.484304 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.484270    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"]
Nov 03 04:17:18.568667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.556483    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtms\" (UniqueName: \"kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.581610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.577138    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:18.581934 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.580904    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.582372 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.582353    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.582614 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.582595    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.591615 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:17:18.580999    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-serving-cert: secret "maas-api-e2e-inf-b-3c1cec-serving-cert" not found
Nov 03 04:17:18.637433 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:17:18.637359    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls podName:668a3983-44bd-4f91-954d-7c501263b172 nodeName:}" failed. No retries permitted until 2026-11-03 04:17:19.137329927 +0000 UTC m=+4530.368222328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls") pod "maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" (UID: "668a3983-44bd-4f91-954d-7c501263b172") : secret "maas-api-e2e-inf-b-3c1cec-serving-cert" not found
Nov 03 04:17:18.645449 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.592593    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.653123 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.653095    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.677369 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.677332    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtms\" (UniqueName: \"kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:18.806579 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.806188    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:18.955042 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:18.954937    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:19.209299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.209209    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:19.211980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.211953    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\") pod \"maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:19.267394 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.267361    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:19.290039 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.290009    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:19.420666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.420632    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:19.668559 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.668520    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"]
Nov 03 04:17:19.669852 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:17:19.669828    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668a3983_44bd_4f91_954d_7c501263b172.slice/crio-bb37c566d109fea6638a4afeee112906aaa81116c068464a1f98836057053e93 WatchSource:0}: Error finding container bb37c566d109fea6638a4afeee112906aaa81116c068464a1f98836057053e93: Status 404 returned error can't find the container with id bb37c566d109fea6638a4afeee112906aaa81116c068464a1f98836057053e93
Nov 03 04:17:19.671346 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.671325    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:17:19.915133 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:19.915099    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" event={"ID":"668a3983-44bd-4f91-954d-7c501263b172","Type":"ContainerStarted","Data":"bb37c566d109fea6638a4afeee112906aaa81116c068464a1f98836057053e93"}
Nov 03 04:17:20.923808 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:20.923773    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" event={"ID":"668a3983-44bd-4f91-954d-7c501263b172","Type":"ContainerStarted","Data":"926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02"}
Nov 03 04:17:20.924233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:20.923847    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:20.984611 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:17:20.984579    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:17:21.046254 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:21.046193    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" podStartSLOduration=2.157948913 podStartE2EDuration="3.046179077s" totalImagesPullingTime="888.230164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:17:21.044086458 +0000 UTC m=+4532.274978876" watchObservedRunningTime="2026-11-03 04:17:21.046179077 +0000 UTC m=+4532.277071515"
Nov 03 04:17:23.740216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:23.740183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:26.938875 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:26.938846    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:17:28.676570 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:28.676518    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:29.612803 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:29.612760    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:29.778943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:29.778908    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:30.100108 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:30.100020    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:33.779051 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:33.779020    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:36.351610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:36.351576    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:36.505166 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:36.505133    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:36.652219 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:36.652180    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:38.723405 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:38.723378    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:41.613447 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:41.613411    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"]
Nov 03 04:17:41.806637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:41.806599    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:43.742882 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:43.742852    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:44.531722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:44.531637    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:45.412900 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:45.412863    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:45.573400 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:45.573357    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:46.135702 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:46.135667    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:47.885256 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:47.885223    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:48.925797 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:48.925759    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:17:49.157244 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:49.157214    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:50.985023 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:17:50.984986    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:17:52.756987 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:52.756951    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:54.131803 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:54.131761    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:54.323997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:54.323964    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:55.759012 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:55.758979    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"]
Nov 03 04:17:56.003466 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.003438    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:56.146165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.146091    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:17:56.635145 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.635112    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main" containerID="cri-o://2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2" gracePeriod=45
Nov 03 04:17:56.895902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.895847    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 04:17:56.999606 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.999574    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7\" (UniqueName: \"kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7\") pod \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") "
Nov 03 04:17:56.999784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.999619    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache\") pod \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") "
Nov 03 04:17:56.999861 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.999777    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm\") pod \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") "
Nov 03 04:17:56.999861 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.999844    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache" pod "2b96b271-e20b-4178-a7c5-8290d0aed2b5" (UID: "2b96b271-e20b-4178-a7c5-8290d0aed2b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:56.999985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.999867    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs\") pod \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") "
Nov 03 04:17:56.999985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:56.999925    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home\") pod \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") "
Nov 03 04:17:57.000087 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.000070    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir\") pod \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\" (UID: \"2b96b271-e20b-4178-a7c5-8290d0aed2b5\") "
Nov 03 04:17:57.000216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.000192    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home" pod "2b96b271-e20b-4178-a7c5-8290d0aed2b5" (UID: "2b96b271-e20b-4178-a7c5-8290d0aed2b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:57.000352 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.000321    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir" pod "2b96b271-e20b-4178-a7c5-8290d0aed2b5" (UID: "2b96b271-e20b-4178-a7c5-8290d0aed2b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:57.000805 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.000781    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:57.000899 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.000810    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:57.000899 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.000826    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:57.002119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.002091    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm" pod "2b96b271-e20b-4178-a7c5-8290d0aed2b5" (UID: "2b96b271-e20b-4178-a7c5-8290d0aed2b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:17:57.002208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.002196    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs" pod "2b96b271-e20b-4178-a7c5-8290d0aed2b5" (UID: "2b96b271-e20b-4178-a7c5-8290d0aed2b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:17:57.002251 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.002218    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7" pod "2b96b271-e20b-4178-a7c5-8290d0aed2b5" (UID: "2b96b271-e20b-4178-a7c5-8290d0aed2b5"). InnerVolumeSpecName "kube-api-access-t97f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:17:57.102799 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.102767    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97f7\" (UniqueName: \"kubernetes.io/projected/2b96b271-e20b-4178-a7c5-8290d0aed2b5-kube-api-access-t97f7\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:57.103036 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.103020    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b96b271-e20b-4178-a7c5-8290d0aed2b5-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:57.103137 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.103126    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b96b271-e20b-4178-a7c5-8290d0aed2b5-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:17:57.181498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.181349    5515 generic.go:356] "Generic (PLEG): container finished" podID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerID="2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2" exitCode=0
Nov 03 04:17:57.181498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.181395    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"
Nov 03 04:17:57.181498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.181437    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" event={"ID":"2b96b271-e20b-4178-a7c5-8290d0aed2b5","Type":"ContainerDied","Data":"2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2"}
Nov 03 04:17:57.181498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.181493    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv" event={"ID":"2b96b271-e20b-4178-a7c5-8290d0aed2b5","Type":"ContainerDied","Data":"613f0b1bd3b19aad277deeda497cf8173cba68e2ce1285025d6b21f4024316da"}
Nov 03 04:17:57.181915 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.181517    5515 scope.go:122] "RemoveContainer" containerID="2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2"
Nov 03 04:17:57.194004 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.193980    5515 scope.go:122] "RemoveContainer" containerID="2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2"
Nov 03 04:17:57.194326 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:17:57.194306    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2\": container with ID starting with 2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2 not found: ID does not exist" containerID="2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2"
Nov 03 04:17:57.194394 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.194335    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2"} err="failed to get container status \"2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2\": rpc error: code = NotFound desc = could not find container \"2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2\": container with ID starting with 2ea82c0cd6e5dd21cce4d0285ad7eb74293142b05f7242ff496e61d607368ba2 not found: ID does not exist"
Nov 03 04:17:57.380379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.380344    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"]
Nov 03 04:17:57.384090 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:57.384063    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-68b670/distinct-2-w3-68b670-kserve-b9d959b84-5mwzv"]
Nov 03 04:17:58.895018 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:58.894987    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" path="/var/lib/kubelet/pods/2b96b271-e20b-4178-a7c5-8290d0aed2b5/volumes"
Nov 03 04:17:59.486250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:17:59.486223    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:00.452005 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:00.451971    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:00.760743 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:00.760653    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz" podUID="8bf262dc-2456-41aa-a3d1-5bcd7126583d" containerName="payload-pre-processing" containerID="cri-o://71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2" gracePeriod=25
Nov 03 04:18:01.035797 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.035774    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 04:18:01.166822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.166790    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh\" (UniqueName: \"kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh\") pod \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") "
Nov 03 04:18:01.167019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.166838    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume\") pod \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\" (UID: \"8bf262dc-2456-41aa-a3d1-5bcd7126583d\") "
Nov 03 04:18:01.167237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.167213    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume" pod "8bf262dc-2456-41aa-a3d1-5bcd7126583d" (UID: "8bf262dc-2456-41aa-a3d1-5bcd7126583d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:18:01.167350 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.167334    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf262dc-2456-41aa-a3d1-5bcd7126583d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:18:01.169070 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.169046    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh" pod "8bf262dc-2456-41aa-a3d1-5bcd7126583d" (UID: "8bf262dc-2456-41aa-a3d1-5bcd7126583d"). InnerVolumeSpecName "kube-api-access-gq6lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:18:01.213082 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.213051    5515 generic.go:356] "Generic (PLEG): container finished" podID="8bf262dc-2456-41aa-a3d1-5bcd7126583d" containerID="71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2" exitCode=0
Nov 03 04:18:01.213242 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.213099    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz" event={"ID":"8bf262dc-2456-41aa-a3d1-5bcd7126583d","Type":"ContainerDied","Data":"71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2"}
Nov 03 04:18:01.213242 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.213135    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz" event={"ID":"8bf262dc-2456-41aa-a3d1-5bcd7126583d","Type":"ContainerDied","Data":"95f303a2ea687e513eb7aae88c733230787e4c10d1f95fd83356615a21d4526c"}
Nov 03 04:18:01.213242 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.213154    5515 scope.go:122] "RemoveContainer" containerID="71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2"
Nov 03 04:18:01.213242 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.213236    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"
Nov 03 04:18:01.227280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.227259    5515 scope.go:122] "RemoveContainer" containerID="71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2"
Nov 03 04:18:01.227567 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:18:01.227534    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2\": container with ID starting with 71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2 not found: ID does not exist" containerID="71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2"
Nov 03 04:18:01.227652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.227590    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2"} err="failed to get container status \"71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2\": rpc error: code = NotFound desc = could not find container \"71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2\": container with ID starting with 71eb76e55d19ca5e8225b1d440f5c8be13cb0dc8b5a9e9a9c11b0eb72d4689f2 not found: ID does not exist"
Nov 03 04:18:01.268267 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.268214    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq6lh\" (UniqueName: \"kubernetes.io/projected/8bf262dc-2456-41aa-a3d1-5bcd7126583d-kube-api-access-gq6lh\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:18:01.381108 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.381077    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"]
Nov 03 04:18:01.384708 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:01.384668    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-68b670-548c98b847-72brz"]
Nov 03 04:18:02.895183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:02.895150    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf262dc-2456-41aa-a3d1-5bcd7126583d" path="/var/lib/kubelet/pods/8bf262dc-2456-41aa-a3d1-5bcd7126583d/volumes"
Nov 03 04:18:03.971078 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:03.971040    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:04.737209 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:04.737158    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:06.988487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:06.988456    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:09.650506 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:09.650474    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:14.339015 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:14.338983    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:14.647288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:14.647255    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:17.825512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:17.825477    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:17.986574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:17.986486    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:19.987311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:19.987278    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:20.129999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:20.129965    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:20.984791 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:18:20.984758    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:18:21.761526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:21.761499    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:24.923024 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:24.922940    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:26.089660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:26.089626    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:27.698774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:27.698745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:29.820308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:29.820270    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:34.403802 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:34.403772    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:18:34.747476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:34.747392    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:38.389355 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:38.389320    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:39.747214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:39.747179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:41.808441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:41.808407    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:44.645139 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:44.645109    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:47.536883 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:47.536850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:47.728527 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:47.728492    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:49.530138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:49.530081    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:50.984752 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:18:50.984714    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:18:54.263032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:54.262992    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:54.727069 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:54.727031    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:18:59.691574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:18:59.689915    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:00.833002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:00.832966    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:00.982106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:00.982077    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:01.134221 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:01.134172    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:04.831089 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:04.831053    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:06.549986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:06.549954    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:09.722829 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:09.722795    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:13.963239 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:13.963195    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:14.638340 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:14.638299    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:15.756100 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:15.756061    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:16.952245 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:16.952201    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:19.497516 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:19.497464    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:20.111802 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:20.111767    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:20.619370 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:20.619330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:20.820436 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:20.820406    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:20.968240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:20.968156    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:20.992116 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:19:20.992079    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:19:23.637778 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:23.637714    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:24.733639 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:24.733606    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:24.938475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:24.938442    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:26.122158 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:26.122120    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:26.482639 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:26.482543    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:19:29.679234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:29.679190    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:31.141145 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:31.141113    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:31.341992 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:31.341942    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:36.124327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:36.124295    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:40.947930 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:40.947899    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:45.955872 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:45.955841    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:50.195021 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:50.194993    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:50.984307 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:19:50.984268    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:19:51.954846 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:51.954815    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:56.820807 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:56.820727    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:58.659459 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:58.647441    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:19:58.814196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:19:58.814167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:00.268430 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:00.268395    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:00.850880 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:00.850845    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:01.696712 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:01.696669    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:01.971411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:01.971330    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:07.245533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:07.245497    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:07.429696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:07.429651    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:08.654589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:08.648118    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:12.613601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:12.613534    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:13.282333 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:13.282299    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:20:14.983980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:14.983944    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:15.684328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:15.684297    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:17.528315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:17.528284    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:20.984618 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:20:20.984576    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:20:21.817469 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:21.817421    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:23.631834 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:23.631802    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:28.524675 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:28.524632    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:29.459589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:29.459557    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:29.606114 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:29.606078    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:31.075522 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:31.075487    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:31.613218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:31.613183    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:32.122449 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:32.122423    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:32.283440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:32.283405    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:32.708561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:32.708507    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:33.869984 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:33.869922    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:38.946820 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:38.946786    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:39.157959 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:39.157896    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:39.486911 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:39.486877    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:40.515813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:40.515784    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:40.654071 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:40.654013    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:44.025315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:44.025280    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:44.546782 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:44.546702    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:46.153687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:46.153656    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:46.854230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:46.854196    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"]
Nov 03 04:20:47.136157 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:47.136125    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:47.275895 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:47.275850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:49.202190 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:49.202158    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:50.984385 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:20:50.984345    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:20:51.855220 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:51.855163    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing" containerID="cri-o://7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9" gracePeriod=25
Nov 03 04:20:52.025535 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:52.025497    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:20:52.341286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:52.341256    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:54.323253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:54.323218    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:20:59.241885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:20:59.241854    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:00.773126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:00.773090    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:02.026127 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:02.026088    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:21:04.127374 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:04.127343    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:06.904933 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:06.904899    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:07.047699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:07.047670    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:07.285685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:07.285593    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:08.349033 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:08.348987    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:21:09.272475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:09.272440    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:12.025724 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:21:12.025697    5515 logging.go:55] [core] [Channel #13257 SubChannel #13258]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.181:50052", ServerName: "10.134.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.181:50052: connect: connection refused"
Nov 03 04:21:12.117794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.117728    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 04:21:12.181276 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.181251    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume\") pod \"415ade92-351b-4c3f-b432-3d43af90403d\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") "
Nov 03 04:21:12.181450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.181421    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx\" (UniqueName: \"kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx\") pod \"415ade92-351b-4c3f-b432-3d43af90403d\" (UID: \"415ade92-351b-4c3f-b432-3d43af90403d\") "
Nov 03 04:21:12.181673 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.181650    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume" pod "415ade92-351b-4c3f-b432-3d43af90403d" (UID: "415ade92-351b-4c3f-b432-3d43af90403d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:21:12.181850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.181835    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/415ade92-351b-4c3f-b432-3d43af90403d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:12.183702 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.183679    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx" pod "415ade92-351b-4c3f-b432-3d43af90403d" (UID: "415ade92-351b-4c3f-b432-3d43af90403d"). InnerVolumeSpecName "kube-api-access-cl2dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:21:12.283327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.283294    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl2dx\" (UniqueName: \"kubernetes.io/projected/415ade92-351b-4c3f-b432-3d43af90403d-kube-api-access-cl2dx\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:12.659677 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.659641    5515 generic.go:356] "Generic (PLEG): container finished" podID="415ade92-351b-4c3f-b432-3d43af90403d" containerID="7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9" exitCode=0
Nov 03 04:21:12.659870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.659688    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"
Nov 03 04:21:12.659870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.659724    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" event={"ID":"415ade92-351b-4c3f-b432-3d43af90403d","Type":"ContainerDied","Data":"7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9"}
Nov 03 04:21:12.659870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.659784    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" event={"ID":"415ade92-351b-4c3f-b432-3d43af90403d","Type":"ContainerDied","Data":"ca76232579a21952f1da753685b15d7f0a923d4a31188bcad824694c3aae94f6"}
Nov 03 04:21:12.659870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.659810    5515 scope.go:122] "RemoveContainer" containerID="7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9"
Nov 03 04:21:12.670447 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.670429    5515 scope.go:122] "RemoveContainer" containerID="7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9"
Nov 03 04:21:12.670790 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:12.670772    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9\": container with ID starting with 7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9 not found: ID does not exist" containerID="7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9"
Nov 03 04:21:12.670853 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.670801    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9"} err="failed to get container status \"7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9\": rpc error: code = NotFound desc = could not find container \"7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9\": container with ID starting with 7db4e94ca526c952d9d2b6adcd7dc5515a301672581c2b585992779cd5fa34a9 not found: ID does not exist"
Nov 03 04:21:12.826989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.826958    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"]
Nov 03 04:21:12.877495 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.877464    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l"]
Nov 03 04:21:12.895139 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:12.895100    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415ade92-351b-4c3f-b432-3d43af90403d" path="/var/lib/kubelet/pods/415ade92-351b-4c3f-b432-3d43af90403d/volumes"
Nov 03 04:21:13.026119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:13.026045    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-10120d-596db45999-vkm2l" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.181:50052\" within 1s: context deadline exceeded"
Nov 03 04:21:13.746323 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:13.746289    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"]
Nov 03 04:21:13.746526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:13.746502    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" podUID="668a3983-44bd-4f91-954d-7c501263b172" containerName="maas-api" containerID="cri-o://926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02" gracePeriod=30
Nov 03 04:21:13.846019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:13.845977    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:14.059784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.059746    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:14.062819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.062800    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:21:14.115129 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.115107    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\") pod \"668a3983-44bd-4f91-954d-7c501263b172\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") "
Nov 03 04:21:14.115285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.115273    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca\") pod \"668a3983-44bd-4f91-954d-7c501263b172\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") "
Nov 03 04:21:14.115337 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.115311    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms\" (UniqueName: \"kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms\") pod \"668a3983-44bd-4f91-954d-7c501263b172\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") "
Nov 03 04:21:14.115374 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.115345    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs\") pod \"668a3983-44bd-4f91-954d-7c501263b172\" (UID: \"668a3983-44bd-4f91-954d-7c501263b172\") "
Nov 03 04:21:14.115733 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.115697    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca" pod "668a3983-44bd-4f91-954d-7c501263b172" (UID: "668a3983-44bd-4f91-954d-7c501263b172"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:21:14.115867 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.115793    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3983-44bd-4f91-954d-7c501263b172-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:14.117460 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.117434    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls" pod "668a3983-44bd-4f91-954d-7c501263b172" (UID: "668a3983-44bd-4f91-954d-7c501263b172"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:21:14.117804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.117781    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs" pod "668a3983-44bd-4f91-954d-7c501263b172" (UID: "668a3983-44bd-4f91-954d-7c501263b172"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:21:14.117873 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.117848    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms" pod "668a3983-44bd-4f91-954d-7c501263b172" (UID: "668a3983-44bd-4f91-954d-7c501263b172"). InnerVolumeSpecName "kube-api-access-6xtms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:21:14.217107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.217075    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:14.217107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.217104    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtms\" (UniqueName: \"kubernetes.io/projected/668a3983-44bd-4f91-954d-7c501263b172-kube-api-access-6xtms\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:14.217107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.217114    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3983-44bd-4f91-954d-7c501263b172-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:14.683605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.680500    5515 generic.go:356] "Generic (PLEG): container finished" podID="668a3983-44bd-4f91-954d-7c501263b172" containerID="926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02" exitCode=0
Nov 03 04:21:14.683605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.680726    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" event={"ID":"668a3983-44bd-4f91-954d-7c501263b172","Type":"ContainerDied","Data":"926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02"}
Nov 03 04:21:14.683605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.680783    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7" event={"ID":"668a3983-44bd-4f91-954d-7c501263b172","Type":"ContainerDied","Data":"bb37c566d109fea6638a4afeee112906aaa81116c068464a1f98836057053e93"}
Nov 03 04:21:14.683605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.680813    5515 scope.go:122] "RemoveContainer" containerID="926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02"
Nov 03 04:21:14.683605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.680959    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"
Nov 03 04:21:14.784575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.781628    5515 scope.go:122] "RemoveContainer" containerID="926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02"
Nov 03 04:21:14.784575 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:14.782838    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02\": container with ID starting with 926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02 not found: ID does not exist" containerID="926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02"
Nov 03 04:21:14.784575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.782878    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02"} err="failed to get container status \"926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02\": rpc error: code = NotFound desc = could not find container \"926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02\": container with ID starting with 926fa15699e73e92b6504c7ae14dddcb62f0769ee20cef8e2063ef6a786a7e02 not found: ID does not exist"
Nov 03 04:21:14.906870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.906838    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"]
Nov 03 04:21:14.958116 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.958031    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3c1cec-5b779fd678-q76t7"]
Nov 03 04:21:14.961002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.960974    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"]
Nov 03 04:21:14.961244 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:14.961217    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" podUID="f2361be4-77ae-456b-99f6-80f3c5aa0aed" containerName="maas-api" containerID="cri-o://d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc" gracePeriod=30
Nov 03 04:21:15.013707 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.013669    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:15.280828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.280800    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 04:21:15.354402 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.354361    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\") pod \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") "
Nov 03 04:21:15.354742 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.354516    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca\") pod \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") "
Nov 03 04:21:15.354742 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.354630    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs\") pod \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") "
Nov 03 04:21:15.354872 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.354782    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs\" (UniqueName: \"kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs\") pod \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\" (UID: \"f2361be4-77ae-456b-99f6-80f3c5aa0aed\") "
Nov 03 04:21:15.356880 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.356843    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca" pod "f2361be4-77ae-456b-99f6-80f3c5aa0aed" (UID: "f2361be4-77ae-456b-99f6-80f3c5aa0aed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:21:15.369595 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.366483    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs" pod "f2361be4-77ae-456b-99f6-80f3c5aa0aed" (UID: "f2361be4-77ae-456b-99f6-80f3c5aa0aed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:21:15.369595 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.366983    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls" pod "f2361be4-77ae-456b-99f6-80f3c5aa0aed" (UID: "f2361be4-77ae-456b-99f6-80f3c5aa0aed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:21:15.369595 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.367018    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:15.369595 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.367710    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs" pod "f2361be4-77ae-456b-99f6-80f3c5aa0aed" (UID: "f2361be4-77ae-456b-99f6-80f3c5aa0aed"). InnerVolumeSpecName "kube-api-access-pszvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:21:15.457196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.457156    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2361be4-77ae-456b-99f6-80f3c5aa0aed-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:15.457196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.457194    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:15.457406 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.457213    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pszvs\" (UniqueName: \"kubernetes.io/projected/f2361be4-77ae-456b-99f6-80f3c5aa0aed-kube-api-access-pszvs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:15.457406 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.457228    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2361be4-77ae-456b-99f6-80f3c5aa0aed-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:15.690087 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.690056    5515 generic.go:356] "Generic (PLEG): container finished" podID="f2361be4-77ae-456b-99f6-80f3c5aa0aed" containerID="d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc" exitCode=0
Nov 03 04:21:15.690288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.690105    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"
Nov 03 04:21:15.690288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.690106    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" event={"ID":"f2361be4-77ae-456b-99f6-80f3c5aa0aed","Type":"ContainerDied","Data":"d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc"}
Nov 03 04:21:15.690288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.690222    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd" event={"ID":"f2361be4-77ae-456b-99f6-80f3c5aa0aed","Type":"ContainerDied","Data":"2cb332c7e911e7f8857218732bbb85705cdfdd41f440c57eef091b74f8f185e2"}
Nov 03 04:21:15.690288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.690244    5515 scope.go:122] "RemoveContainer" containerID="d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc"
Nov 03 04:21:15.701142 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.701123    5515 scope.go:122] "RemoveContainer" containerID="d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc"
Nov 03 04:21:15.701399 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:15.701377    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc\": container with ID starting with d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc not found: ID does not exist" containerID="d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc"
Nov 03 04:21:15.701466 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.701405    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc"} err="failed to get container status \"d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc\": rpc error: code = NotFound desc = could not find container \"d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc\": container with ID starting with d2795bbd65dfb1324635ff6d6dc8b191d2e3f4677ce4ab7ed32a573ad5daecbc not found: ID does not exist"
Nov 03 04:21:15.911130 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.909652    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"]
Nov 03 04:21:15.917598 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:15.913328    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-f3e02b-564dcbdd5-9fcjd"]
Nov 03 04:21:16.902020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:16.901984    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668a3983-44bd-4f91-954d-7c501263b172" path="/var/lib/kubelet/pods/668a3983-44bd-4f91-954d-7c501263b172/volumes"
Nov 03 04:21:16.902529 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:16.902513    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2361be4-77ae-456b-99f6-80f3c5aa0aed" path="/var/lib/kubelet/pods/f2361be4-77ae-456b-99f6-80f3c5aa0aed/volumes"
Nov 03 04:21:17.070987 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:17.070937    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:18.455743 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:18.455710    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:18.603080 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:18.603049    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:20.000255 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:20.000222    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:20.455227 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:20.455198    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:20.603353 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:20.603318    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:20.984981 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:20.984940    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:21:21.587775 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:21.587745    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:21.832599 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:21.832536    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:22.065431 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:22.065397    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:22.558940 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:22.558853    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:22.714454 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:22.714427    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:23.519972 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:23.519942    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:27.123465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:27.123433    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:28.936342 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:28.936308    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:30.673107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:30.673074    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:30.679162 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:30.679139    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"]
Nov 03 04:21:30.792851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:30.792819    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"]
Nov 03 04:21:31.040308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:31.040238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:31.182857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:31.182824    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:34.224448 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:34.224415    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:35.580315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:35.580281    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:35.679807 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:35.679771    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" podUID="39620b2e-b2d5-4162-8115-d40e81939478" containerName="payload-processing" containerID="cri-o://2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931" gracePeriod=25
Nov 03 04:21:35.722335 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:35.722304    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:35.793960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:35.793931    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerName="payload-pre-processing" containerID="cri-o://339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f" gracePeriod=25
Nov 03 04:21:36.547273 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:36.547244    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:37.177857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:37.177825    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:37.716245 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:37.716211    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:38.338850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:38.338816    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:38.564758 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:38.564729    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:39.628662 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:39.628629    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:42.395423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:42.395389    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" podUID="39620b2e-b2d5-4162-8115-d40e81939478" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:21:44.408214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:44.408177    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:21:44.609927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:44.609898    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:44.754292 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:44.754256    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:44.903852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:44.903769    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:45.387628 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:45.387598    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:49.938682 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:49.938654    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:50.384301 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:50.384276    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:21:50.386605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:50.386585    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:21:50.427523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:50.427500    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:21:50.429661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:50.429640    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:21:50.983365 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:50.983331    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:21:51.434985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:51.434900    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:52.395186 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:52.395141    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" podUID="39620b2e-b2d5-4162-8115-d40e81939478" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:21:54.408375 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:54.408337    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:21:54.831703 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:54.831623    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:21:55.965390 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:55.965367    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 04:21:56.019182 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.019144    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk\" (UniqueName: \"kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk\") pod \"39620b2e-b2d5-4162-8115-d40e81939478\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") "
Nov 03 04:21:56.019353 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.019325    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume\") pod \"39620b2e-b2d5-4162-8115-d40e81939478\" (UID: \"39620b2e-b2d5-4162-8115-d40e81939478\") "
Nov 03 04:21:56.019813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.019789    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume" pod "39620b2e-b2d5-4162-8115-d40e81939478" (UID: "39620b2e-b2d5-4162-8115-d40e81939478"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:21:56.020096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.019907    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39620b2e-b2d5-4162-8115-d40e81939478-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:56.021828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.021794    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk" pod "39620b2e-b2d5-4162-8115-d40e81939478" (UID: "39620b2e-b2d5-4162-8115-d40e81939478"). InnerVolumeSpecName "kube-api-access-dgsqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:21:56.044310 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.044289    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 04:21:56.061376 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.061354    5515 generic.go:356] "Generic (PLEG): container finished" podID="39620b2e-b2d5-4162-8115-d40e81939478" containerID="2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931" exitCode=0
Nov 03 04:21:56.061473 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.061404    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"
Nov 03 04:21:56.061473 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.061438    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" event={"ID":"39620b2e-b2d5-4162-8115-d40e81939478","Type":"ContainerDied","Data":"2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931"}
Nov 03 04:21:56.061602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.061494    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml" event={"ID":"39620b2e-b2d5-4162-8115-d40e81939478","Type":"ContainerDied","Data":"b64604a4cedf02c57530d2d276d4c8ca8af3a1a0e9b60f90cfafc3653d31d975"}
Nov 03 04:21:56.061602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.061522    5515 scope.go:122] "RemoveContainer" containerID="2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931"
Nov 03 04:21:56.062906 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.062882    5515 generic.go:356] "Generic (PLEG): container finished" podID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerID="339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f" exitCode=0
Nov 03 04:21:56.063012 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.062929    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"
Nov 03 04:21:56.063012 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.062984    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" event={"ID":"3a63b964-d5a5-489d-becb-6c7fd95ec450","Type":"ContainerDied","Data":"339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f"}
Nov 03 04:21:56.063102 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.063034    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9" event={"ID":"3a63b964-d5a5-489d-becb-6c7fd95ec450","Type":"ContainerDied","Data":"540f87e7dde213b3c322f715935cb4e9ef43a395bd63750160dd2c751cb6064b"}
Nov 03 04:21:56.072261 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.072245    5515 scope.go:122] "RemoveContainer" containerID="2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931"
Nov 03 04:21:56.072536 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:56.072519    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931\": container with ID starting with 2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931 not found: ID does not exist" containerID="2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931"
Nov 03 04:21:56.072628 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.072563    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931"} err="failed to get container status \"2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931\": rpc error: code = NotFound desc = could not find container \"2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931\": container with ID starting with 2545fcc163a96c327af1d83e28e11f0c2a7bb719a92e66abd52e41ce2d175931 not found: ID does not exist"
Nov 03 04:21:56.072628 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.072597    5515 scope.go:122] "RemoveContainer" containerID="339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f"
Nov 03 04:21:56.081064 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.081047    5515 scope.go:122] "RemoveContainer" containerID="339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f"
Nov 03 04:21:56.081316 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:21:56.081299    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f\": container with ID starting with 339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f not found: ID does not exist" containerID="339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f"
Nov 03 04:21:56.081373 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.081322    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f"} err="failed to get container status \"339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f\": rpc error: code = NotFound desc = could not find container \"339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f\": container with ID starting with 339a2f032080d856a24b70a4de4d9e4f137335ac4c1f62a6f31c9004c39e9a0f not found: ID does not exist"
Nov 03 04:21:56.120647 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.120622    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume\") pod \"3a63b964-d5a5-489d-becb-6c7fd95ec450\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") "
Nov 03 04:21:56.120814 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.120653    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j\" (UniqueName: \"kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j\") pod \"3a63b964-d5a5-489d-becb-6c7fd95ec450\" (UID: \"3a63b964-d5a5-489d-becb-6c7fd95ec450\") "
Nov 03 04:21:56.120992 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.120970    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume" pod "3a63b964-d5a5-489d-becb-6c7fd95ec450" (UID: "3a63b964-d5a5-489d-becb-6c7fd95ec450"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:21:56.121068 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.120989    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgsqk\" (UniqueName: \"kubernetes.io/projected/39620b2e-b2d5-4162-8115-d40e81939478-kube-api-access-dgsqk\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:56.123037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.123010    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j" pod "3a63b964-d5a5-489d-becb-6c7fd95ec450" (UID: "3a63b964-d5a5-489d-becb-6c7fd95ec450"). InnerVolumeSpecName "kube-api-access-92j8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:21:56.221675 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.221617    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a63b964-d5a5-489d-becb-6c7fd95ec450-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:56.221675 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.221638    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92j8j\" (UniqueName: \"kubernetes.io/projected/3a63b964-d5a5-489d-becb-6c7fd95ec450-kube-api-access-92j8j\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:21:56.238183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.238156    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"]
Nov 03 04:21:56.244953 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.244931    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-7ed2d6-5b5d44d5d7-5mpml"]
Nov 03 04:21:56.485945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.485856    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"]
Nov 03 04:21:56.489989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.489966    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-7ed2d6-56f6858dcf-lmph9"]
Nov 03 04:21:56.625963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.625936    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:21:56.894525 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.894495    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39620b2e-b2d5-4162-8115-d40e81939478" path="/var/lib/kubelet/pods/39620b2e-b2d5-4162-8115-d40e81939478/volumes"
Nov 03 04:21:56.894916 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:56.894902    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" path="/var/lib/kubelet/pods/3a63b964-d5a5-489d-becb-6c7fd95ec450/volumes"
Nov 03 04:21:59.728929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:21:59.728896    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:04.623989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:04.623956    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:09.455061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:09.455000    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:14.332430 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:14.332396    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:15.716562 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:15.716527    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:18.584291 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:18.584235    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:19.751876 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:19.751836    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:20.984315 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:20.984278    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:22:21.533685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:21.533650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:25.879326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:25.879294    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:26.029288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:26.029252    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:26.482719 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:26.482687    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:26.971876 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:26.971840    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:27.121730 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:27.121693    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:27.467146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:27.467110    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:28.011350 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:28.011323    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:28.015014 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:28.014985    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"]
Nov 03 04:22:28.015206 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:28.015182    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" containerID="cri-o://56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b" gracePeriod=30
Nov 03 04:22:28.572511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:28.572478    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:28.952486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:28.952452    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:29.485754 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:29.485723    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"]
Nov 03 04:22:29.486138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:29.485909    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy" containerID="cri-o://0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a" gracePeriod=30
Nov 03 04:22:29.497785 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:29.497761    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:29.989841 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:29.989808    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:30.381618 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:30.381582    5515 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 04:22:30.381793 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:30.381649    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 04:22:31.819865 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:31.819827    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:33.299280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.299223    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 04:22:33.356359 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.356326    5515 generic.go:356] "Generic (PLEG): container finished" podID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerID="56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b" exitCode=0
Nov 03 04:22:33.356519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.356377    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"
Nov 03 04:22:33.356519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.356408    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" event={"ID":"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff","Type":"ContainerDied","Data":"56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b"}
Nov 03 04:22:33.356519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.356470    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692" event={"ID":"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff","Type":"ContainerDied","Data":"71a3d5abf3e3740e2c035e8726992a4343efebc3e739cf1773c4c94f3b5d7e46"}
Nov 03 04:22:33.356519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.356501    5515 scope.go:122] "RemoveContainer" containerID="56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b"
Nov 03 04:22:33.365685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.365667    5515 scope.go:122] "RemoveContainer" containerID="bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"
Nov 03 04:22:33.374136 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.374121    5515 scope.go:122] "RemoveContainer" containerID="56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b"
Nov 03 04:22:33.374419 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:33.374401    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b\": container with ID starting with 56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b not found: ID does not exist" containerID="56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b"
Nov 03 04:22:33.374486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.374427    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b"} err="failed to get container status \"56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b\": rpc error: code = NotFound desc = could not find container \"56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b\": container with ID starting with 56034b1c63970805cd2bce564faa6ebe097c5abcf6c01dc926a61ba5b3fd459b not found: ID does not exist"
Nov 03 04:22:33.374486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.374446    5515 scope.go:122] "RemoveContainer" containerID="bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"
Nov 03 04:22:33.374809 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:33.374790    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231\": container with ID starting with bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231 not found: ID does not exist" containerID="bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"
Nov 03 04:22:33.374861 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.374818    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231"} err="failed to get container status \"bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231\": rpc error: code = NotFound desc = could not find container \"bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231\": container with ID starting with bf48a16b275b31af0b72444d8c7fefc29116031195aeab3d68fbbc48280c6231 not found: ID does not exist"
Nov 03 04:22:33.381748 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.381728    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.381852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.381807    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.381852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.381834    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.381958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.381886    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.381958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.381920    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.381958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.381951    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.382126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382006    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.382126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382060    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.382126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382083    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:33.382304 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382131    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:33.382304 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382147    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs\") pod \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\" (UID: \"6ce77fd3-d98f-4ac4-b8ba-562e561c8dff\") "
Nov 03 04:22:33.382430 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382360    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:33.382496 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382470    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:33.382607 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382585    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:22:33.382674 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382648    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.382724 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382675    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.382724 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382689    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.382724 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.382705    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.384337 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.384307    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:22:33.384434 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.384388    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "kube-api-access-m6hmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:33.384503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.384450    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:33.384644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.384627    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy" pod "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" (UID: "6ce77fd3-d98f-4ac4-b8ba-562e561c8dff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:33.483734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.483699    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hmd\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-kube-api-access-m6hmd\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.483734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.483734    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.483989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.483750    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.483989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.483766    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.483989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.483780    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:33.531175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.531146    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"]
Nov 03 04:22:33.537801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.537773    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"]
Nov 03 04:22:33.558971 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.558892    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"]
Nov 03 04:22:33.631513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.631487    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:33.998850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:33.998813    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"]
Nov 03 04:22:34.005895 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.005869    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9daccc-openshift-default-d8c6c56bf-dm692"]
Nov 03 04:22:34.750634 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.750583    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 04:22:34.801786 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.801760    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.801903 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.801871    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.801965 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.801905    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.801965 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.801952    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.802072 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.801986    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.802072 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802024    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.802168 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802106    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.802168 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802157    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.802285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802221    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket\") pod \"b9710500-d6bf-45c1-8724-8dcf08b046a5\" (UID: \"b9710500-d6bf-45c1-8724-8dcf08b046a5\") "
Nov 03 04:22:34.802977 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802660    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:34.802977 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802686    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:34.802977 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802730    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:22:34.802977 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.802821    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:34.803456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.803064    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:34.805832 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.805807    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:22:34.805832 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.805820    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:34.805963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.805927    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:34.805963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.805932    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2" pod "b9710500-d6bf-45c1-8724-8dcf08b046a5" (UID: "b9710500-d6bf-45c1-8724-8dcf08b046a5"). InnerVolumeSpecName "kube-api-access-h9rg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:34.897459 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.897419    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" path="/var/lib/kubelet/pods/6ce77fd3-d98f-4ac4-b8ba-562e561c8dff/volumes"
Nov 03 04:22:34.903827 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903797    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903831    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rg2\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-kube-api-access-h9rg2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903846    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903858    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903870    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903883    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903895    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903907    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9710500-d6bf-45c1-8724-8dcf08b046a5-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.903967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.903919    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9710500-d6bf-45c1-8724-8dcf08b046a5-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:34.943573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:34.943537    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"]
Nov 03 04:22:35.115609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.115578    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:35.370349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.370266    5515 generic.go:356] "Generic (PLEG): container finished" podID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerID="0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a" exitCode=0
Nov 03 04:22:35.370349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.370320    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"
Nov 03 04:22:35.370349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.370316    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" event={"ID":"b9710500-d6bf-45c1-8724-8dcf08b046a5","Type":"ContainerDied","Data":"0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a"}
Nov 03 04:22:35.370672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.370392    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8" event={"ID":"b9710500-d6bf-45c1-8724-8dcf08b046a5","Type":"ContainerDied","Data":"edc5ce6fdd118e4a437c9195127d449f5f9511de882b2868af2196d45683c8c3"}
Nov 03 04:22:35.370672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.370407    5515 scope.go:122] "RemoveContainer" containerID="0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a"
Nov 03 04:22:35.379967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.379945    5515 scope.go:122] "RemoveContainer" containerID="b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"
Nov 03 04:22:35.389817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.389785    5515 scope.go:122] "RemoveContainer" containerID="0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a"
Nov 03 04:22:35.390075 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:35.390057    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a\": container with ID starting with 0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a not found: ID does not exist" containerID="0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a"
Nov 03 04:22:35.390125 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.390086    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a"} err="failed to get container status \"0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a\": rpc error: code = NotFound desc = could not find container \"0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a\": container with ID starting with 0a8f13767d136a58987abeb638e66ad1d60827ce30a37a88828257011069778a not found: ID does not exist"
Nov 03 04:22:35.390125 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.390104    5515 scope.go:122] "RemoveContainer" containerID="b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"
Nov 03 04:22:35.390374 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:35.390357    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb\": container with ID starting with b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb not found: ID does not exist" containerID="b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"
Nov 03 04:22:35.390419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.390392    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb"} err="failed to get container status \"b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb\": rpc error: code = NotFound desc = could not find container \"b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb\": container with ID starting with b582adbb09df5486c8a7ec64bd9e2b4467208c2e4645b08d662db70c9c00a8eb not found: ID does not exist"
Nov 03 04:22:35.538869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.538835    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"]
Nov 03 04:22:35.589127 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.589094    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7ed2d6-openshift-default-786cb48569-qj9p8"]
Nov 03 04:22:35.779740 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:35.779655    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:36.742622 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:36.742590    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:36.893767 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:36.893733    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" path="/var/lib/kubelet/pods/b9710500-d6bf-45c1-8724-8dcf08b046a5/volumes"
Nov 03 04:22:37.446916 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:37.446885    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:41.299686 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:41.299640    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:22:41.571663 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:41.571578    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:42.514422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:42.514383    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:43.203037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:43.202999    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:43.354924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:43.354891    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:44.707347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:44.707315    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:45.571363 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:45.571331    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:45.716066 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:45.716035    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:46.921127 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:46.921097    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:48.553816 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.553780    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" containerID="cri-o://ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17" gracePeriod=45
Nov 03 04:22:48.559619 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.559558    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main" containerID="cri-o://b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6" gracePeriod=45
Nov 03 04:22:48.584145 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.584088    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" containerID="cri-o://e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313" gracePeriod=45
Nov 03 04:22:48.653788 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.653754    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": dial tcp 10.134.0.69:8000: connect: connection refused"
Nov 03 04:22:48.686079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.686047    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused"
Nov 03 04:22:48.880510 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.880468    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 04:22:48.886668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.886644    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 04:22:48.906786 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.906767    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 04:22:48.920804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.920779    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:48.999005 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.998971    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir\") pod \"131361f6-42d9-40a0-adb7-927c17ecf9b0\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") "
Nov 03 04:22:48.999233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999022    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs\") pod \"1b35c0e4-d1a3-4528-924b-1f095099ee53\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") "
Nov 03 04:22:48.999233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999067    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs\") pod \"b11e6b5c-300c-48e3-a427-446eddd74f52\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") "
Nov 03 04:22:48.999233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999117    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache\") pod \"b11e6b5c-300c-48e3-a427-446eddd74f52\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") "
Nov 03 04:22:48.999233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999160    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir\") pod \"1b35c0e4-d1a3-4528-924b-1f095099ee53\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") "
Nov 03 04:22:48.999233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999223    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm\") pod \"b11e6b5c-300c-48e3-a427-446eddd74f52\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999278    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache\") pod \"1b35c0e4-d1a3-4528-924b-1f095099ee53\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999314    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5\" (UniqueName: \"kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5\") pod \"1b35c0e4-d1a3-4528-924b-1f095099ee53\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999380    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm\") pod \"131361f6-42d9-40a0-adb7-927c17ecf9b0\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999406    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home\") pod \"131361f6-42d9-40a0-adb7-927c17ecf9b0\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999438    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home\") pod \"b11e6b5c-300c-48e3-a427-446eddd74f52\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999471    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home\") pod \"1b35c0e4-d1a3-4528-924b-1f095099ee53\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999528    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm\" (UniqueName: \"kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm\") pod \"b11e6b5c-300c-48e3-a427-446eddd74f52\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") "
Nov 03 04:22:48.999699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999675    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir\") pod \"b11e6b5c-300c-48e3-a427-446eddd74f52\" (UID: \"b11e6b5c-300c-48e3-a427-446eddd74f52\") "
Nov 03 04:22:49.000299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999714    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz\" (UniqueName: \"kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz\") pod \"131361f6-42d9-40a0-adb7-927c17ecf9b0\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") "
Nov 03 04:22:49.000299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999748    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache\") pod \"131361f6-42d9-40a0-adb7-927c17ecf9b0\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") "
Nov 03 04:22:49.000299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999801    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs\") pod \"131361f6-42d9-40a0-adb7-927c17ecf9b0\" (UID: \"131361f6-42d9-40a0-adb7-927c17ecf9b0\") "
Nov 03 04:22:49.000299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:48.999832    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm\") pod \"1b35c0e4-d1a3-4528-924b-1f095099ee53\" (UID: \"1b35c0e4-d1a3-4528-924b-1f095099ee53\") "
Nov 03 04:22:49.000911 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.000887    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache" pod "b11e6b5c-300c-48e3-a427-446eddd74f52" (UID: "b11e6b5c-300c-48e3-a427-446eddd74f52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.001140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.000981    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home" pod "131361f6-42d9-40a0-adb7-927c17ecf9b0" (UID: "131361f6-42d9-40a0-adb7-927c17ecf9b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.001286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.001139    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home" pod "b11e6b5c-300c-48e3-a427-446eddd74f52" (UID: "b11e6b5c-300c-48e3-a427-446eddd74f52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.001411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.001298    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home" pod "1b35c0e4-d1a3-4528-924b-1f095099ee53" (UID: "1b35c0e4-d1a3-4528-924b-1f095099ee53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.002193 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.000393    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir" pod "1b35c0e4-d1a3-4528-924b-1f095099ee53" (UID: "1b35c0e4-d1a3-4528-924b-1f095099ee53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.002484 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.002458    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir" pod "131361f6-42d9-40a0-adb7-927c17ecf9b0" (UID: "131361f6-42d9-40a0-adb7-927c17ecf9b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.002981 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.002955    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm" pod "131361f6-42d9-40a0-adb7-927c17ecf9b0" (UID: "131361f6-42d9-40a0-adb7-927c17ecf9b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.003679 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.003480    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache" pod "131361f6-42d9-40a0-adb7-927c17ecf9b0" (UID: "131361f6-42d9-40a0-adb7-927c17ecf9b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.004569 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.003828    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache" pod "1b35c0e4-d1a3-4528-924b-1f095099ee53" (UID: "1b35c0e4-d1a3-4528-924b-1f095099ee53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.004569 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.004148    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm" pod "b11e6b5c-300c-48e3-a427-446eddd74f52" (UID: "b11e6b5c-300c-48e3-a427-446eddd74f52"). InnerVolumeSpecName "kube-api-access-797zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:49.005425 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.004776    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir" pod "b11e6b5c-300c-48e3-a427-446eddd74f52" (UID: "b11e6b5c-300c-48e3-a427-446eddd74f52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.005425 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.004786    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5" pod "1b35c0e4-d1a3-4528-924b-1f095099ee53" (UID: "1b35c0e4-d1a3-4528-924b-1f095099ee53"). InnerVolumeSpecName "kube-api-access-9mps5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:49.006240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.005853    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs" pod "1b35c0e4-d1a3-4528-924b-1f095099ee53" (UID: "1b35c0e4-d1a3-4528-924b-1f095099ee53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:22:49.007816 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.007790    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm" pod "b11e6b5c-300c-48e3-a427-446eddd74f52" (UID: "b11e6b5c-300c-48e3-a427-446eddd74f52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.008895 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.008876    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm" pod "1b35c0e4-d1a3-4528-924b-1f095099ee53" (UID: "1b35c0e4-d1a3-4528-924b-1f095099ee53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:49.008979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.008888    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs" pod "131361f6-42d9-40a0-adb7-927c17ecf9b0" (UID: "131361f6-42d9-40a0-adb7-927c17ecf9b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:22:49.008979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.008900    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs" pod "b11e6b5c-300c-48e3-a427-446eddd74f52" (UID: "b11e6b5c-300c-48e3-a427-446eddd74f52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:22:49.009106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.009088    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz" pod "131361f6-42d9-40a0-adb7-927c17ecf9b0" (UID: "131361f6-42d9-40a0-adb7-927c17ecf9b0"). InnerVolumeSpecName "kube-api-access-ntpmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101652    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101686    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101697    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mps5\" (UniqueName: \"kubernetes.io/projected/1b35c0e4-d1a3-4528-924b-1f095099ee53-kube-api-access-9mps5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101706    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101714    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101721    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101728    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101735    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797zm\" (UniqueName: \"kubernetes.io/projected/b11e6b5c-300c-48e3-a427-446eddd74f52-kube-api-access-797zm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.101734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101743    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101752    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntpmz\" (UniqueName: \"kubernetes.io/projected/131361f6-42d9-40a0-adb7-927c17ecf9b0-kube-api-access-ntpmz\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101761    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101770    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131361f6-42d9-40a0-adb7-927c17ecf9b0-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101779    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101786    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131361f6-42d9-40a0-adb7-927c17ecf9b0-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101793    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b35c0e4-d1a3-4528-924b-1f095099ee53-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101802    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11e6b5c-300c-48e3-a427-446eddd74f52-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101809    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11e6b5c-300c-48e3-a427-446eddd74f52-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.102175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.101817    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b35c0e4-d1a3-4528-924b-1f095099ee53-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:49.488489 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.488459    5515 generic.go:356] "Generic (PLEG): container finished" podID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerID="e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313" exitCode=0
Nov 03 04:22:49.488693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.488502    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"
Nov 03 04:22:49.488693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.488563    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" event={"ID":"b11e6b5c-300c-48e3-a427-446eddd74f52","Type":"ContainerDied","Data":"e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313"}
Nov 03 04:22:49.488693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.488608    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8" event={"ID":"b11e6b5c-300c-48e3-a427-446eddd74f52","Type":"ContainerDied","Data":"a267fb6265b559240beba24afcffab9d4401ca352e6a845b7a9ccbc7c7cd485c"}
Nov 03 04:22:49.488693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.488632    5515 scope.go:122] "RemoveContainer" containerID="e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313"
Nov 03 04:22:49.490325 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.490253    5515 generic.go:356] "Generic (PLEG): container finished" podID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerID="b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6" exitCode=0
Nov 03 04:22:49.490325 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.490287    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"
Nov 03 04:22:49.490539 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.490333    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" event={"ID":"131361f6-42d9-40a0-adb7-927c17ecf9b0","Type":"ContainerDied","Data":"b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6"}
Nov 03 04:22:49.490539 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.490368    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl" event={"ID":"131361f6-42d9-40a0-adb7-927c17ecf9b0","Type":"ContainerDied","Data":"3f016dca5e384a1e130a2ba92ebac4a3e22225d873e5def991748e2fe502ab88"}
Nov 03 04:22:49.491838 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.491813    5515 generic.go:356] "Generic (PLEG): container finished" podID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerID="ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17" exitCode=0
Nov 03 04:22:49.491976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.491847    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" event={"ID":"1b35c0e4-d1a3-4528-924b-1f095099ee53","Type":"ContainerDied","Data":"ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17"}
Nov 03 04:22:49.491976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.491863    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"
Nov 03 04:22:49.491976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.491885    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l" event={"ID":"1b35c0e4-d1a3-4528-924b-1f095099ee53","Type":"ContainerDied","Data":"73cda1da7bc970ea44542a33dcb64a3335889c92af4b36554aadd5a7088a8bf3"}
Nov 03 04:22:49.499874 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.499857    5515 scope.go:122] "RemoveContainer" containerID="e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313"
Nov 03 04:22:49.500271 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:49.500143    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313\": container with ID starting with e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313 not found: ID does not exist" containerID="e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313"
Nov 03 04:22:49.500271 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.500175    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313"} err="failed to get container status \"e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313\": rpc error: code = NotFound desc = could not find container \"e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313\": container with ID starting with e4cc0e4b34cfe5e4559ae7dac8ad6c529cdf54ea56a8e039861530399474a313 not found: ID does not exist"
Nov 03 04:22:49.500271 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.500196    5515 scope.go:122] "RemoveContainer" containerID="b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6"
Nov 03 04:22:49.508918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.508899    5515 scope.go:122] "RemoveContainer" containerID="b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6"
Nov 03 04:22:49.509157 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:49.509136    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6\": container with ID starting with b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6 not found: ID does not exist" containerID="b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6"
Nov 03 04:22:49.509213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.509165    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6"} err="failed to get container status \"b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6\": rpc error: code = NotFound desc = could not find container \"b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6\": container with ID starting with b1389b98cc40ab16c100afb9c87ceaab15a8842c514e909e7d8609be4734f1c6 not found: ID does not exist"
Nov 03 04:22:49.509213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.509181    5515 scope.go:122] "RemoveContainer" containerID="ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17"
Nov 03 04:22:49.517198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.517179    5515 scope.go:122] "RemoveContainer" containerID="ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17"
Nov 03 04:22:49.517452 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:49.517434    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17\": container with ID starting with ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17 not found: ID does not exist" containerID="ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17"
Nov 03 04:22:49.517499 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.517458    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17"} err="failed to get container status \"ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17\": rpc error: code = NotFound desc = could not find container \"ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17\": container with ID starting with ef551bee9c082cd373efc670e1917ed34adc8cc0b4ad7e361ff98d40b5d4dd17 not found: ID does not exist"
Nov 03 04:22:49.655174 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.655142    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"]
Nov 03 04:22:49.661475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.661449    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/simulator-w1-5a6246-kserve-755d547c68-979q8"]
Nov 03 04:22:49.772093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.772009    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"]
Nov 03 04:22:49.778773 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.778749    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-2-w1-5a6246-kserve-78756cb55f-v9s2l"]
Nov 03 04:22:49.889224 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.889200    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"]
Nov 03 04:22:49.892627 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.892604    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5a6246/distinct-w1-5a6246-kserve-569b85464-bl7fl"]
Nov 03 04:22:49.964622 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:49.964589    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main" containerID="cri-o://3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd" gracePeriod=45
Nov 03 04:22:50.225658 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.225637    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 04:22:50.313951 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.313923    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs\") pod \"8fb747ca-ec06-45af-986d-dc4582cd3583\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") "
Nov 03 04:22:50.314113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.313980    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home\") pod \"8fb747ca-ec06-45af-986d-dc4582cd3583\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") "
Nov 03 04:22:50.314191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314132    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m\" (UniqueName: \"kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m\") pod \"8fb747ca-ec06-45af-986d-dc4582cd3583\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") "
Nov 03 04:22:50.314272 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314187    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache\") pod \"8fb747ca-ec06-45af-986d-dc4582cd3583\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") "
Nov 03 04:22:50.314272 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314240    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home" pod "8fb747ca-ec06-45af-986d-dc4582cd3583" (UID: "8fb747ca-ec06-45af-986d-dc4582cd3583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:50.314403 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314309    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm\") pod \"8fb747ca-ec06-45af-986d-dc4582cd3583\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") "
Nov 03 04:22:50.314403 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314347    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir\") pod \"8fb747ca-ec06-45af-986d-dc4582cd3583\" (UID: \"8fb747ca-ec06-45af-986d-dc4582cd3583\") "
Nov 03 04:22:50.314518 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314417    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache" pod "8fb747ca-ec06-45af-986d-dc4582cd3583" (UID: "8fb747ca-ec06-45af-986d-dc4582cd3583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:50.314706 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314683    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir" pod "8fb747ca-ec06-45af-986d-dc4582cd3583" (UID: "8fb747ca-ec06-45af-986d-dc4582cd3583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:50.314874 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314857    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:50.314946 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314881    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:50.314946 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.314895    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:50.316432 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.316412    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m" pod "8fb747ca-ec06-45af-986d-dc4582cd3583" (UID: "8fb747ca-ec06-45af-986d-dc4582cd3583"). InnerVolumeSpecName "kube-api-access-gd57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:22:50.316848 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.316829    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm" pod "8fb747ca-ec06-45af-986d-dc4582cd3583" (UID: "8fb747ca-ec06-45af-986d-dc4582cd3583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:22:50.316924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.316851    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs" pod "8fb747ca-ec06-45af-986d-dc4582cd3583" (UID: "8fb747ca-ec06-45af-986d-dc4582cd3583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:22:50.415804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.415769    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd57m\" (UniqueName: \"kubernetes.io/projected/8fb747ca-ec06-45af-986d-dc4582cd3583-kube-api-access-gd57m\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:50.415804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.415799    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb747ca-ec06-45af-986d-dc4582cd3583-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:50.416002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.415815    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb747ca-ec06-45af-986d-dc4582cd3583-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:22:50.500106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.500075    5515 generic.go:356] "Generic (PLEG): container finished" podID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerID="3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd" exitCode=0
Nov 03 04:22:50.500250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.500129    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"
Nov 03 04:22:50.500250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.500153    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" event={"ID":"8fb747ca-ec06-45af-986d-dc4582cd3583","Type":"ContainerDied","Data":"3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd"}
Nov 03 04:22:50.500250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.500190    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5" event={"ID":"8fb747ca-ec06-45af-986d-dc4582cd3583","Type":"ContainerDied","Data":"7312c895cd593bc68cc6f780e403634092d0bdbc7727e06bcfd5a3fc5bd260ff"}
Nov 03 04:22:50.500250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.500209    5515 scope.go:122] "RemoveContainer" containerID="3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd"
Nov 03 04:22:50.509421 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.509404    5515 scope.go:122] "RemoveContainer" containerID="3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd"
Nov 03 04:22:50.509701 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:50.509682    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd\": container with ID starting with 3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd not found: ID does not exist" containerID="3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd"
Nov 03 04:22:50.509759 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.509710    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd"} err="failed to get container status \"3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd\": rpc error: code = NotFound desc = could not find container \"3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd\": container with ID starting with 3062f8b3910235bad739115fcb265362e3b5e5bc2633145a8ebaed2797aa9edd not found: ID does not exist"
Nov 03 04:22:50.667120 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.667042    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"]
Nov 03 04:22:50.676368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.676340    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b1ad4e/distinct-2-w4-b1ad4e-kserve-b58fd967c-ps6s5"]
Nov 03 04:22:50.893687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.893651    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" path="/var/lib/kubelet/pods/131361f6-42d9-40a0-adb7-927c17ecf9b0/volumes"
Nov 03 04:22:50.894108 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.894094    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" path="/var/lib/kubelet/pods/1b35c0e4-d1a3-4528-924b-1f095099ee53/volumes"
Nov 03 04:22:50.894473 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.894461    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" path="/var/lib/kubelet/pods/8fb747ca-ec06-45af-986d-dc4582cd3583/volumes"
Nov 03 04:22:50.895191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:50.895175    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" path="/var/lib/kubelet/pods/b11e6b5c-300c-48e3-a427-446eddd74f52/volumes"
Nov 03 04:22:50.984979 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:22:50.984893    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:22:51.030799 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:51.030764    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:51.820931 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:51.820823    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:55.040007 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:55.039921    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:56.735984 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:56.735951    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:56.883863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:56.883834    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:22:57.179415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:22:57.179387    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:01.578175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:01.578146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:06.499053 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:06.499016    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:08.826239 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:08.826203    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:09.801684 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:09.801640    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:09.948678 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:09.948642    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:11.398092 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:11.398059    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:11.650209 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:11.650128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:11.802693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:11.802660    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:13.359965 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:13.359935    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:14.935794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:14.935761    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:16.070052 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:16.070023    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:16.339172 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:16.339092    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:16.559863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:16.559830    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:17.007354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:17.007311    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"]
Nov 03 04:23:17.371255 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:17.371226    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:17.519226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:17.519189    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:19.056167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:19.056128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:19.205385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:19.205355    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:20.984840 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:20.984806    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:23:21.202843 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:21.202814    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:21.366148 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:21.366103    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:21.694183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:21.694100    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:22.008238 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:22.008148    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerName="payload-processing" containerID="cri-o://029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4" gracePeriod=25
Nov 03 04:23:22.511453 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:22.511417    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:23.237963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:23.237921    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:23:24.489099 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:24.489060    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:23:25.998343 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:25.998309    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:27.824625 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:27.824591    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:29.766382 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:29.766351    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:29.914691 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:29.914661    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:31.344486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:31.344451    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:31.874066 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:31.874039    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:31.880681 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:31.880657    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"]
Nov 03 04:23:31.880885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:31.880867    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy" containerID="cri-o://04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453" gracePeriod=30
Nov 03 04:23:32.300948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:32.300862    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:34.234767 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:34.234722    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:34.488497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:34.488399    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:23:37.131058 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.131036    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 04:23:37.177199 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177136    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177353 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177169    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177353 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177319    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177382    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177462    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177630 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177506    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177630 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177600    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177630 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177612    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:37.177801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177632    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177699    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket\") pod \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\" (UID: \"f6c56464-7dbd-411a-a63f-14fec9ea0c58\") "
Nov 03 04:23:37.177920 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177798    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:37.178001 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.177980    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:23:37.178285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178259    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:37.178368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178282    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:37.178520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178474    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.178520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178501    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.178520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178516    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.178520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178531    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.178520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.178572    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.179662 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.179636    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:23:37.179759 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.179675    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "kube-api-access-qphsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:23:37.179759 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.179698    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:37.180076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.180058    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo" pod "f6c56464-7dbd-411a-a63f-14fec9ea0c58" (UID: "f6c56464-7dbd-411a-a63f-14fec9ea0c58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:23:37.280182 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.280151    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphsr\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-kube-api-access-qphsr\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.280182 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.280178    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.280386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.280189    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.280386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.280200    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c56464-7dbd-411a-a63f-14fec9ea0c58-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:37.781527 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.781494    5515 generic.go:356] "Generic (PLEG): container finished" podID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerID="04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453" exitCode=0
Nov 03 04:23:37.781737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.781542    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"
Nov 03 04:23:37.781737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.781582    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" event={"ID":"f6c56464-7dbd-411a-a63f-14fec9ea0c58","Type":"ContainerDied","Data":"04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453"}
Nov 03 04:23:37.781737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.781624    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn" event={"ID":"f6c56464-7dbd-411a-a63f-14fec9ea0c58","Type":"ContainerDied","Data":"298014adb46c6ae94538b94f01ac61f04d529c9df1481ccfba8046a0856a9e82"}
Nov 03 04:23:37.781737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.781645    5515 scope.go:122] "RemoveContainer" containerID="04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453"
Nov 03 04:23:37.791896 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.791882    5515 scope.go:122] "RemoveContainer" containerID="647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"
Nov 03 04:23:37.800236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.800216    5515 scope.go:122] "RemoveContainer" containerID="04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453"
Nov 03 04:23:37.800526 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:37.800507    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453\": container with ID starting with 04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453 not found: ID does not exist" containerID="04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453"
Nov 03 04:23:37.800623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.800534    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453"} err="failed to get container status \"04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453\": rpc error: code = NotFound desc = could not find container \"04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453\": container with ID starting with 04c2001fcc62f992314cb3314ef431b52348830527556cd7ad22f364287b6453 not found: ID does not exist"
Nov 03 04:23:37.800623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.800571    5515 scope.go:122] "RemoveContainer" containerID="647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"
Nov 03 04:23:37.800846 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:37.800828    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799\": container with ID starting with 647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799 not found: ID does not exist" containerID="647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"
Nov 03 04:23:37.800910 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.800857    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799"} err="failed to get container status \"647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799\": rpc error: code = NotFound desc = could not find container \"647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799\": container with ID starting with 647cc69e451db24c80a5a352f0e48bd8f01568145de123d6a3270d9e667fe799 not found: ID does not exist"
Nov 03 04:23:37.948712 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.948672    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"]
Nov 03 04:23:37.999272 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:37.999239    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5a6246-openshift-default-755c68b7b7-dn9mn"]
Nov 03 04:23:38.894159 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:38.894125    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" path="/var/lib/kubelet/pods/f6c56464-7dbd-411a-a63f-14fec9ea0c58/volumes"
Nov 03 04:23:39.329620 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:39.329505    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:41.435345 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:41.435309    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:41.670806 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:41.670770    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:42.258794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.258764    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 04:23:42.336910 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.336848    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m\" (UniqueName: \"kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m\") pod \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") "
Nov 03 04:23:42.337026 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.337004    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume\") pod \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\" (UID: \"cc0a5e36-a7c0-44d5-b987-a79a18bac90b\") "
Nov 03 04:23:42.337334 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.337313    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume" pod "cc0a5e36-a7c0-44d5-b987-a79a18bac90b" (UID: "cc0a5e36-a7c0-44d5-b987-a79a18bac90b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:23:42.339106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.339085    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m" pod "cc0a5e36-a7c0-44d5-b987-a79a18bac90b" (UID: "cc0a5e36-a7c0-44d5-b987-a79a18bac90b"). InnerVolumeSpecName "kube-api-access-j7q9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:23:42.438761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.438733    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:42.438761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.438759    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7q9m\" (UniqueName: \"kubernetes.io/projected/cc0a5e36-a7c0-44d5-b987-a79a18bac90b-kube-api-access-j7q9m\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:42.809970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.809934    5515 generic.go:356] "Generic (PLEG): container finished" podID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerID="029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4" exitCode=0
Nov 03 04:23:42.810173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.809984    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"
Nov 03 04:23:42.810173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.810020    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" event={"ID":"cc0a5e36-a7c0-44d5-b987-a79a18bac90b","Type":"ContainerDied","Data":"029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4"}
Nov 03 04:23:42.810173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.810073    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942" event={"ID":"cc0a5e36-a7c0-44d5-b987-a79a18bac90b","Type":"ContainerDied","Data":"5fe72f75202663ed1b081f7f99e81826373f9aab48eddb8b284616b5a9c41115"}
Nov 03 04:23:42.810173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.810099    5515 scope.go:122] "RemoveContainer" containerID="029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4"
Nov 03 04:23:42.819712 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.819693    5515 scope.go:122] "RemoveContainer" containerID="029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4"
Nov 03 04:23:42.819981 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:42.819964    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4\": container with ID starting with 029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4 not found: ID does not exist" containerID="029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4"
Nov 03 04:23:42.820040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.819991    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4"} err="failed to get container status \"029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4\": rpc error: code = NotFound desc = could not find container \"029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4\": container with ID starting with 029cb0e4fea7382b3f8c4c03a177b082339c9610849320e3fb440c7d643c0bb4 not found: ID does not exist"
Nov 03 04:23:42.979046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.979016    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"]
Nov 03 04:23:42.985076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:42.985055    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-efa9bf-775644c6cf-gj942"]
Nov 03 04:23:43.071758 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:43.071684    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:43.573038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:43.573004    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:43.724641 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:43.713312    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"]
Nov 03 04:23:43.724641 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:43.713615    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" podUID="428d296d-b09c-4d16-8486-7f12791481bb" containerName="maas-api" containerID="cri-o://6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd" gracePeriod=30
Nov 03 04:23:43.798368 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:43.798336    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:44.034731 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.034704    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 04:23:44.168810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.168721    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs\") pod \"428d296d-b09c-4d16-8486-7f12791481bb\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") "
Nov 03 04:23:44.168810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.168785    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\") pod \"428d296d-b09c-4d16-8486-7f12791481bb\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") "
Nov 03 04:23:44.169064 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.168889    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5\" (UniqueName: \"kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5\") pod \"428d296d-b09c-4d16-8486-7f12791481bb\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") "
Nov 03 04:23:44.169064 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.168936    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca\") pod \"428d296d-b09c-4d16-8486-7f12791481bb\" (UID: \"428d296d-b09c-4d16-8486-7f12791481bb\") "
Nov 03 04:23:44.170380 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.170073    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca" pod "428d296d-b09c-4d16-8486-7f12791481bb" (UID: "428d296d-b09c-4d16-8486-7f12791481bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:23:44.178397 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.178369    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls" pod "428d296d-b09c-4d16-8486-7f12791481bb" (UID: "428d296d-b09c-4d16-8486-7f12791481bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:23:44.179225 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.178533    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5" pod "428d296d-b09c-4d16-8486-7f12791481bb" (UID: "428d296d-b09c-4d16-8486-7f12791481bb"). InnerVolumeSpecName "kube-api-access-ztjh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:23:44.180153 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.180123    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs" pod "428d296d-b09c-4d16-8486-7f12791481bb" (UID: "428d296d-b09c-4d16-8486-7f12791481bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:23:44.286140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.285248    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:44.286140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.285281    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d296d-b09c-4d16-8486-7f12791481bb-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:44.286140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.285298    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjh5\" (UniqueName: \"kubernetes.io/projected/428d296d-b09c-4d16-8486-7f12791481bb-kube-api-access-ztjh5\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:44.286140 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.285314    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d296d-b09c-4d16-8486-7f12791481bb-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:44.822508 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.822472    5515 generic.go:356] "Generic (PLEG): container finished" podID="428d296d-b09c-4d16-8486-7f12791481bb" containerID="6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd" exitCode=0
Nov 03 04:23:44.822995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.822522    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" event={"ID":"428d296d-b09c-4d16-8486-7f12791481bb","Type":"ContainerDied","Data":"6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd"}
Nov 03 04:23:44.822995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.822533    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"
Nov 03 04:23:44.822995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.822616    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" event={"ID":"428d296d-b09c-4d16-8486-7f12791481bb","Type":"ContainerDied","Data":"dc5e50ab238d0a355b5fef9d8f0680c17bea08a5c7b997571b910a5d02458176"}
Nov 03 04:23:44.822995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.822640    5515 scope.go:122] "RemoveContainer" containerID="6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd"
Nov 03 04:23:44.832509 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.832489    5515 scope.go:122] "RemoveContainer" containerID="6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd"
Nov 03 04:23:44.832814 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:44.832795    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd\": container with ID starting with 6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd not found: ID does not exist" containerID="6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd"
Nov 03 04:23:44.832875 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.832824    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd"} err="failed to get container status \"6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd\": rpc error: code = NotFound desc = could not find container \"6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd\": container with ID starting with 6fa6b9fbc18ed9831103868b789ac8c0818d47c81ee3ee2757b3f0928dabcdbd not found: ID does not exist"
Nov 03 04:23:44.893462 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.893436    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" path="/var/lib/kubelet/pods/cc0a5e36-a7c0-44d5-b987-a79a18bac90b/volumes"
Nov 03 04:23:44.988746 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:44.988705    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"]
Nov 03 04:23:45.039336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:45.039311    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd"]
Nov 03 04:23:45.776175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:45.776140    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:46.893740 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:46.893709    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428d296d-b09c-4d16-8486-7f12791481bb" path="/var/lib/kubelet/pods/428d296d-b09c-4d16-8486-7f12791481bb/volumes"
Nov 03 04:23:47.014889 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:47.014850    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-efa9bf-785d5bbfcd-h68dd" podUID="428d296d-b09c-4d16-8486-7f12791481bb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.189:8443/health\": context deadline exceeded"
Nov 03 04:23:47.158597 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:47.158501    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:47.308473 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:47.308441    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:48.756365 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:48.756318    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:49.289299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:49.289268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:49.301593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:49.301534    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"]
Nov 03 04:23:49.301811 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:49.301791    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy" containerID="cri-o://b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635" gracePeriod=30
Nov 03 04:23:50.157985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:50.157949    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:50.983664 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:50.983624    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:23:51.120209 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:51.120177    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:54.573925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.573868    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 04:23:54.719779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.719752    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.719943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.719836    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.719943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.719873    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.719945    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.719963    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720000    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720043    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720075    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720136    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs\") pod \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\" (UID: \"cb8a6bb3-970e-4fce-bcad-01233bf889fc\") "
Nov 03 04:23:54.720361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720141    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:54.720361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720228    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:54.720361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720154    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:23:54.720361 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720315    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:54.720631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720448    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:54.720631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720569    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.720631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720595    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.720631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720608    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.720631 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720622    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.721067 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.720635    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.722399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.722375    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:23:54.722530 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.722455    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:23:54.722530 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.722490    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:23:54.722530 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.722492    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh" pod "cb8a6bb3-970e-4fce-bcad-01233bf889fc" (UID: "cb8a6bb3-970e-4fce-bcad-01233bf889fc"). InnerVolumeSpecName "kube-api-access-n9nxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:23:54.821741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.821714    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.821741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.821739    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.821915 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.821748    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9nxh\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-kube-api-access-n9nxh\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.821915 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.821758    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb8a6bb3-970e-4fce-bcad-01233bf889fc-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:23:54.894920 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.894889    5515 generic.go:356] "Generic (PLEG): container finished" podID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerID="b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635" exitCode=0
Nov 03 04:23:54.895096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.894925    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" event={"ID":"cb8a6bb3-970e-4fce-bcad-01233bf889fc","Type":"ContainerDied","Data":"b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635"}
Nov 03 04:23:54.895096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.894964    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"
Nov 03 04:23:54.895096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.894975    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc" event={"ID":"cb8a6bb3-970e-4fce-bcad-01233bf889fc","Type":"ContainerDied","Data":"6edc13000317f84e55548d3d19e5f9921228704181453b9331ceaee1f9ed84c1"}
Nov 03 04:23:54.895096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.894995    5515 scope.go:122] "RemoveContainer" containerID="b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635"
Nov 03 04:23:54.905753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.905736    5515 scope.go:122] "RemoveContainer" containerID="3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"
Nov 03 04:23:54.915298 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.915275    5515 scope.go:122] "RemoveContainer" containerID="b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635"
Nov 03 04:23:54.915598 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:54.915579    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635\": container with ID starting with b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635 not found: ID does not exist" containerID="b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635"
Nov 03 04:23:54.915671 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.915607    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635"} err="failed to get container status \"b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635\": rpc error: code = NotFound desc = could not find container \"b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635\": container with ID starting with b9cd4b97de5f5c99a0171db4d445cb346a25bd4239647ee47b01500022e47635 not found: ID does not exist"
Nov 03 04:23:54.915671 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.915627    5515 scope.go:122] "RemoveContainer" containerID="3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"
Nov 03 04:23:54.915852 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:23:54.915837    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7\": container with ID starting with 3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7 not found: ID does not exist" containerID="3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"
Nov 03 04:23:54.915898 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:54.915855    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7"} err="failed to get container status \"3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7\": rpc error: code = NotFound desc = could not find container \"3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7\": container with ID starting with 3fdc40232c0af40c2b71bda9d69dd5b11cdbcf87dcdf956341c84f2e688221e7 not found: ID does not exist"
Nov 03 04:23:55.064526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:55.064496    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"]
Nov 03 04:23:55.066423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:55.066393    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-efa9bf-openshift-default-56fbc788d5-znchc"]
Nov 03 04:23:55.599428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:55.599394    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"]
Nov 03 04:23:55.703157 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:55.703121    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"]
Nov 03 04:23:55.959566 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:55.959519    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:56.105864 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:56.105823    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:56.249775 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:56.249684    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:56.534423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:56.534340    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:23:56.893939 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:23:56.893909    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" path="/var/lib/kubelet/pods/cb8a6bb3-970e-4fce-bcad-01233bf889fc/volumes"
Nov 03 04:24:00.600069 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:00.600027    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerName="payload-processing" containerID="cri-o://b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73" gracePeriod=25
Nov 03 04:24:00.704147 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:00.704106    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" containerName="payload-pre-processing" containerID="cri-o://e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6" gracePeriod=25
Nov 03 04:24:01.330214 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:01.330182    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:02.902878 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:02.902845    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:03.771976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:03.771940    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:24:03.772197 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:03.771951    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:24:05.923499 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:05.923466    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:24:06.222912 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:06.222835    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:08.017093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.017057    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"]
Nov 03 04:24:08.017685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.017318    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy" containerID="cri-o://5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31" gracePeriod=30
Nov 03 04:24:08.067474 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.067446    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"]
Nov 03 04:24:08.067684 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.067664    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" containerID="cri-o://b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823" gracePeriod=30
Nov 03 04:24:08.069250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.069230    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"]
Nov 03 04:24:08.069426 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.069393    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" containerID="cri-o://1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82" gracePeriod=30
Nov 03 04:24:08.118469 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.118436    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"]
Nov 03 04:24:08.118695 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:08.118656    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" containerID="cri-o://6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24" gracePeriod=30
Nov 03 04:24:10.494395 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:10.494349    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4t4rg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 04:24:10.494809 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:10.494416    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 04:24:11.139844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:11.139771    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:12.397448 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:12.397372    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:12.540465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:12.540413    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:13.007036 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.007001    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c9ztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 04:24:13.007237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.007054    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 04:24:13.105498 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.105476    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-rzzkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 04:24:13.105634 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.105536    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 04:24:13.336962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.336937    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:24:13.400302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.400272    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 04:24:13.427240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427114    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.427240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427193    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.427589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427562    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.427732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427608    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.427732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427683    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.427732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427717    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.427908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427765    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.427908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427838    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.427908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427866    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.428093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427906    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.428093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427940    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.428093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.427977    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.428093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.428009    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.428093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.428062    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.428381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.428130    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.428381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.428186    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.428381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.428224    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket\") pod \"a2b5226a-d141-496c-8a53-1a29728f7590\" (UID: \"a2b5226a-d141-496c-8a53-1a29728f7590\") "
Nov 03 04:24:13.428381 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.428283    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs\") pod \"e9e843ff-8d4f-4276-9db2-83febf7e039b\" (UID: \"e9e843ff-8d4f-4276-9db2-83febf7e039b\") "
Nov 03 04:24:13.429660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.429632    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.430769 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.430257    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.430769 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.430450    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.430929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.430827    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.430929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.430849    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.431056 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.431036    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.432086 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.431424    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.432086 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.431514    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.432086 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.431823    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:24:13.435051 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.434832    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:24:13.437296 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437272    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.437296 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437285    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.437511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437306    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "kube-api-access-6f7f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.437511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437384    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo" pod "a2b5226a-d141-496c-8a53-1a29728f7590" (UID: "a2b5226a-d141-496c-8a53-1a29728f7590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:24:13.437511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437403    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.437511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437406    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:24:13.437511 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.437500    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.439256 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.439231    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm" pod "e9e843ff-8d4f-4276-9db2-83febf7e039b" (UID: "e9e843ff-8d4f-4276-9db2-83febf7e039b"). InnerVolumeSpecName "kube-api-access-crmbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.446623 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.446602    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:24:13.448900 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.448883    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:24:13.529541 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529505    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.529753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529586    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.529753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529647    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.529753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529682    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.529753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529715    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.529753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529743    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529784    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529809    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529847    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529894    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529933    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529981    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529981    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.529992    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530009    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530067    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530108    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530136    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530152    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo\") pod \"c445bad5-3c95-4631-b207-996f78b0d51e\" (UID: \"c445bad5-3c95-4631-b207-996f78b0d51e\") "
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530212    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530294    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530334    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530361    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:24:13.530749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.530387    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket\") pod \"246d8290-3d5f-405a-96bb-60964e3d66c6\" (UID: \"246d8290-3d5f-405a-96bb-60964e3d66c6\") "
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532279    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532310    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532328    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532344    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246d8290-3d5f-405a-96bb-60964e3d66c6-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532356    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532367    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532379    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532406    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2b5226a-d141-496c-8a53-1a29728f7590-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532420    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532433    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532449    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532463    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532478    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2b5226a-d141-496c-8a53-1a29728f7590-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532492    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532507    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532521    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c445bad5-3c95-4631-b207-996f78b0d51e-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532534    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f7f2\" (UniqueName: \"kubernetes.io/projected/a2b5226a-d141-496c-8a53-1a29728f7590-kube-api-access-6f7f2\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532566    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmbm\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-kube-api-access-crmbm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532581    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e843ff-8d4f-4276-9db2-83febf7e039b-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532593    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532606    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532619    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2b5226a-d141-496c-8a53-1a29728f7590-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532634    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.532629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532648    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e843ff-8d4f-4276-9db2-83febf7e039b-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.532281    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.533026    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.533099    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "kube-api-access-h44lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.533315    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.533520    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "kube-api-access-d5424". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.533653    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.534124 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.533715    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.535772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.535729    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.535889 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.535814    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token" pod "c445bad5-3c95-4631-b207-996f78b0d51e" (UID: "c445bad5-3c95-4631-b207-996f78b0d51e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.536258 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.536237    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:13.536589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.536526    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:24:13.537198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.537176    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy" pod "246d8290-3d5f-405a-96bb-60964e3d66c6" (UID: "246d8290-3d5f-405a-96bb-60964e3d66c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:13.634077 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634040    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h44lc\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-kube-api-access-h44lc\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634077 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634076    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634077 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634086    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c445bad5-3c95-4631-b207-996f78b0d51e-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634095    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634105    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c445bad5-3c95-4631-b207-996f78b0d51e-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634117    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c445bad5-3c95-4631-b207-996f78b0d51e-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634130    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634142    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634150    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634159    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246d8290-3d5f-405a-96bb-60964e3d66c6-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634166    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246d8290-3d5f-405a-96bb-60964e3d66c6-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.634408 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.634174    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5424\" (UniqueName: \"kubernetes.io/projected/246d8290-3d5f-405a-96bb-60964e3d66c6-kube-api-access-d5424\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:13.772240 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.772202    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:24:13.772421 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:13.772216    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:24:14.011241 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.011151    5515 generic.go:356] "Generic (PLEG): container finished" podID="c445bad5-3c95-4631-b207-996f78b0d51e" containerID="6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24" exitCode=0
Nov 03 04:24:14.011241 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.011206    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"
Nov 03 04:24:14.011494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.011235    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" event={"ID":"c445bad5-3c95-4631-b207-996f78b0d51e","Type":"ContainerDied","Data":"6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24"}
Nov 03 04:24:14.011494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.011293    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz" event={"ID":"c445bad5-3c95-4631-b207-996f78b0d51e","Type":"ContainerDied","Data":"1654748df0c1c4732129112ea24607ba0e0b87a37c46efb0f3fc0f7e82802a0f"}
Nov 03 04:24:14.011494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.011315    5515 scope.go:122] "RemoveContainer" containerID="6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24"
Nov 03 04:24:14.012725 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.012703    5515 generic.go:356] "Generic (PLEG): container finished" podID="a2b5226a-d141-496c-8a53-1a29728f7590" containerID="b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823" exitCode=0
Nov 03 04:24:14.013019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.012756    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"
Nov 03 04:24:14.013019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.012779    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" event={"ID":"a2b5226a-d141-496c-8a53-1a29728f7590","Type":"ContainerDied","Data":"b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823"}
Nov 03 04:24:14.013019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.012820    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg" event={"ID":"a2b5226a-d141-496c-8a53-1a29728f7590","Type":"ContainerDied","Data":"ca8b9262e0f915c8730f827223defb6a1db0551458aed057f799ab3eb690bd95"}
Nov 03 04:24:14.014220 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.014196    5515 generic.go:356] "Generic (PLEG): container finished" podID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerID="5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31" exitCode=0
Nov 03 04:24:14.014336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.014237    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"
Nov 03 04:24:14.014336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.014274    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" event={"ID":"e9e843ff-8d4f-4276-9db2-83febf7e039b","Type":"ContainerDied","Data":"5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31"}
Nov 03 04:24:14.014336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.014322    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t" event={"ID":"e9e843ff-8d4f-4276-9db2-83febf7e039b","Type":"ContainerDied","Data":"d2a6b3e8ff85903aa0e6173ae22418e82253cd11c91823718f3f13c01b97ebc2"}
Nov 03 04:24:14.015902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.015881    5515 generic.go:356] "Generic (PLEG): container finished" podID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerID="1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82" exitCode=0
Nov 03 04:24:14.016023 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.015930    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"
Nov 03 04:24:14.016023 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.015933    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" event={"ID":"246d8290-3d5f-405a-96bb-60964e3d66c6","Type":"ContainerDied","Data":"1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82"}
Nov 03 04:24:14.016152 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.016047    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp" event={"ID":"246d8290-3d5f-405a-96bb-60964e3d66c6","Type":"ContainerDied","Data":"7258976b63393d29c833862cadd7167014bf1399c4bee40f6f1ec5496a6bdf7c"}
Nov 03 04:24:14.022295 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.022076    5515 scope.go:122] "RemoveContainer" containerID="6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24"
Nov 03 04:24:14.022411 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:14.022394    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24\": container with ID starting with 6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24 not found: ID does not exist" containerID="6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24"
Nov 03 04:24:14.022461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.022419    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24"} err="failed to get container status \"6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24\": rpc error: code = NotFound desc = could not find container \"6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24\": container with ID starting with 6054afea2b23aeb8b41753c3aae4f64b2294e24ddc845c55a2c3abd0aa983e24 not found: ID does not exist"
Nov 03 04:24:14.022461 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.022434    5515 scope.go:122] "RemoveContainer" containerID="b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823"
Nov 03 04:24:14.032084 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.032070    5515 scope.go:122] "RemoveContainer" containerID="b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823"
Nov 03 04:24:14.032301 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:14.032286    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823\": container with ID starting with b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823 not found: ID does not exist" containerID="b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823"
Nov 03 04:24:14.032341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.032307    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823"} err="failed to get container status \"b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823\": rpc error: code = NotFound desc = could not find container \"b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823\": container with ID starting with b8c19ada08d0eb91ea1e47b6747c134af4c0d8cb37e925e0793eee0049bc6823 not found: ID does not exist"
Nov 03 04:24:14.032341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.032329    5515 scope.go:122] "RemoveContainer" containerID="5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31"
Nov 03 04:24:14.040578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.040534    5515 scope.go:122] "RemoveContainer" containerID="5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31"
Nov 03 04:24:14.040809 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:14.040790    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31\": container with ID starting with 5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31 not found: ID does not exist" containerID="5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31"
Nov 03 04:24:14.040885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.040815    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31"} err="failed to get container status \"5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31\": rpc error: code = NotFound desc = could not find container \"5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31\": container with ID starting with 5f1b584eae8eca156b59d6045126a232aafd879142af85d32f181cc6db5bcb31 not found: ID does not exist"
Nov 03 04:24:14.040885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.040829    5515 scope.go:122] "RemoveContainer" containerID="1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82"
Nov 03 04:24:14.068345 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.068328    5515 scope.go:122] "RemoveContainer" containerID="1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82"
Nov 03 04:24:14.068646 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:14.068626    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82\": container with ID starting with 1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82 not found: ID does not exist" containerID="1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82"
Nov 03 04:24:14.068706 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.068653    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82"} err="failed to get container status \"1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82\": rpc error: code = NotFound desc = could not find container \"1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82\": container with ID starting with 1b83b8da540a6dc60ff25bcd6392d3826d212a58d1b2e041564eed0530b41e82 not found: ID does not exist"
Nov 03 04:24:14.179802 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.179763    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"]
Nov 03 04:24:14.184044 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.184022    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c9ztz"]
Nov 03 04:24:14.295610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.295506    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"]
Nov 03 04:24:14.304114 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.304088    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rzzkp"]
Nov 03 04:24:14.415696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.415666    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"]
Nov 03 04:24:14.419583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.419541    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-rbh9t"]
Nov 03 04:24:14.531301 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.531272    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"]
Nov 03 04:24:14.534937 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.534912    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4t4rg"]
Nov 03 04:24:14.894096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.894062    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" path="/var/lib/kubelet/pods/246d8290-3d5f-405a-96bb-60964e3d66c6/volumes"
Nov 03 04:24:14.894520 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.894506    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" path="/var/lib/kubelet/pods/a2b5226a-d141-496c-8a53-1a29728f7590/volumes"
Nov 03 04:24:14.894925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.894912    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" path="/var/lib/kubelet/pods/c445bad5-3c95-4631-b207-996f78b0d51e/volumes"
Nov 03 04:24:14.895633 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:14.895619    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" path="/var/lib/kubelet/pods/e9e843ff-8d4f-4276-9db2-83febf7e039b/volumes"
Nov 03 04:24:16.130340 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:16.130292    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:16.418036 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:16.417949    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:20.862385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:20.862353    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 04:24:20.903298 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:20.903269    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg\" (UniqueName: \"kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg\") pod \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") "
Nov 03 04:24:20.903509 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:20.903363    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume\") pod \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\" (UID: \"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5\") "
Nov 03 04:24:20.903726 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:20.903700    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume" pod "3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" (UID: "3b8434e7-5ef7-40f0-a981-cdbfb1e287b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:24:20.905609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:20.905579    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg" pod "3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" (UID: "3b8434e7-5ef7-40f0-a981-cdbfb1e287b5"). InnerVolumeSpecName "kube-api-access-6cpjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:20.943896 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:20.943872    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 04:24:20.983392 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:20.983356    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:24:21.004678 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.004653    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume\") pod \"b00600f9-826b-48ba-b576-dfe4e299957c\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") "
Nov 03 04:24:21.004800 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.004730    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r\" (UniqueName: \"kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r\") pod \"b00600f9-826b-48ba-b576-dfe4e299957c\" (UID: \"b00600f9-826b-48ba-b576-dfe4e299957c\") "
Nov 03 04:24:21.005079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.005054    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume" pod "b00600f9-826b-48ba-b576-dfe4e299957c" (UID: "b00600f9-826b-48ba-b576-dfe4e299957c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:24:21.005236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.005214    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b00600f9-826b-48ba-b576-dfe4e299957c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:21.005290 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.005242    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpjg\" (UniqueName: \"kubernetes.io/projected/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-kube-api-access-6cpjg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:21.005290 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.005257    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:21.006795 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.006774    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r" pod "b00600f9-826b-48ba-b576-dfe4e299957c" (UID: "b00600f9-826b-48ba-b576-dfe4e299957c"). InnerVolumeSpecName "kube-api-access-jgl7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:21.056487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.056454    5515 generic.go:356] "Generic (PLEG): container finished" podID="b00600f9-826b-48ba-b576-dfe4e299957c" containerID="e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6" exitCode=0
Nov 03 04:24:21.056719 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.056515    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"
Nov 03 04:24:21.056719 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.056616    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" event={"ID":"b00600f9-826b-48ba-b576-dfe4e299957c","Type":"ContainerDied","Data":"e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6"}
Nov 03 04:24:21.066419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.066376    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd" event={"ID":"b00600f9-826b-48ba-b576-dfe4e299957c","Type":"ContainerDied","Data":"d3cd29d1d2b4c5a6f160d8a4ce6cde979fdef5e5416b4ee8e84015f494b1043e"}
Nov 03 04:24:21.067077 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.066593    5515 scope.go:122] "RemoveContainer" containerID="e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6"
Nov 03 04:24:21.076530 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.076499    5515 generic.go:356] "Generic (PLEG): container finished" podID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerID="b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73" exitCode=0
Nov 03 04:24:21.076732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.076597    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"
Nov 03 04:24:21.076732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.076596    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" event={"ID":"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5","Type":"ContainerDied","Data":"b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73"}
Nov 03 04:24:21.076732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.076654    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs" event={"ID":"3b8434e7-5ef7-40f0-a981-cdbfb1e287b5","Type":"ContainerDied","Data":"1ed86ee4342a283b138ee958073363653f0a8154c538c7a68f06b0cce1c43d8a"}
Nov 03 04:24:21.084930 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.084910    5515 scope.go:122] "RemoveContainer" containerID="e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6"
Nov 03 04:24:21.085220 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:21.085195    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6\": container with ID starting with e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6 not found: ID does not exist" containerID="e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6"
Nov 03 04:24:21.085346 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.085232    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6"} err="failed to get container status \"e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6\": rpc error: code = NotFound desc = could not find container \"e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6\": container with ID starting with e462c6e72f6c8978f4e1e524e53980cbd7268d77da13b667e77030cce806e1b6 not found: ID does not exist"
Nov 03 04:24:21.085346 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.085256    5515 scope.go:122] "RemoveContainer" containerID="b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73"
Nov 03 04:24:21.094411 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.094387    5515 scope.go:122] "RemoveContainer" containerID="b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73"
Nov 03 04:24:21.094674 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:21.094655    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73\": container with ID starting with b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73 not found: ID does not exist" containerID="b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73"
Nov 03 04:24:21.094739 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.094682    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73"} err="failed to get container status \"b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73\": rpc error: code = NotFound desc = could not find container \"b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73\": container with ID starting with b83d4f97c11f6dd005b900e5a8edb289800d2414c8175499c0b9d8e3037f0f73 not found: ID does not exist"
Nov 03 04:24:21.105754 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.105733    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgl7r\" (UniqueName: \"kubernetes.io/projected/b00600f9-826b-48ba-b576-dfe4e299957c-kube-api-access-jgl7r\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:21.255245 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.255159    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"]
Nov 03 04:24:21.259638 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.259610    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-286a8c-b8bdd58c9-jvfcs"]
Nov 03 04:24:21.370280 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.370246    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"]
Nov 03 04:24:21.375960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:21.375936    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-286a8c-55999b575-7rncd"]
Nov 03 04:24:22.148375 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:22.148340    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:22.892919 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:22.892883    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" path="/var/lib/kubelet/pods/3b8434e7-5ef7-40f0-a981-cdbfb1e287b5/volumes"
Nov 03 04:24:22.893273 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:22.893259    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" path="/var/lib/kubelet/pods/b00600f9-826b-48ba-b576-dfe4e299957c/volumes"
Nov 03 04:24:25.257265 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:25.257237    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:27.049390 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:27.049355    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:27.204541 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:27.204506    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:28.840329 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:28.840300    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:29.409270 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:29.409236    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:29.902504 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:29.902462    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:32.184527 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:32.184491    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:33.338149 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:33.338089    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:34.517025 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:34.516994    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:24:34.871284 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:34.871252    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"]
Nov 03 04:24:34.921831 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:34.921797    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"]
Nov 03 04:24:35.055482 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:35.055445    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:37.147508 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:37.147479    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:41.161670 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:41.161625    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:42.121064 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:42.121031    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:46.926475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:46.926445    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:47.241075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:47.240990    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:47.927383 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:47.927345    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"]
Nov 03 04:24:48.036718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:48.036681    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"]
Nov 03 04:24:48.284948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:48.284866    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:48.426644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:48.426607    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:49.892700 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:49.892662    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main" containerID="cri-o://292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224" gracePeriod=45
Nov 03 04:24:49.941760 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:49.941722    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main" containerID="cri-o://26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6" gracePeriod=45
Nov 03 04:24:50.188270 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.188247    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 04:24:50.218854 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.218828    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 04:24:50.221972 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.221950    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs\") pod \"a89dd4ba-b869-4902-9cd3-968437773d1f\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") "
Nov 03 04:24:50.222060 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.221987    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache\") pod \"a89dd4ba-b869-4902-9cd3-968437773d1f\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") "
Nov 03 04:24:50.222060 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222050    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home\") pod \"a89dd4ba-b869-4902-9cd3-968437773d1f\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") "
Nov 03 04:24:50.222142 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222077    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm\") pod \"a89dd4ba-b869-4902-9cd3-968437773d1f\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") "
Nov 03 04:24:50.222142 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222136    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir\") pod \"a89dd4ba-b869-4902-9cd3-968437773d1f\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") "
Nov 03 04:24:50.222252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222185    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w\" (UniqueName: \"kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w\") pod \"a89dd4ba-b869-4902-9cd3-968437773d1f\" (UID: \"a89dd4ba-b869-4902-9cd3-968437773d1f\") "
Nov 03 04:24:50.222344 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222315    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache" pod "a89dd4ba-b869-4902-9cd3-968437773d1f" (UID: "a89dd4ba-b869-4902-9cd3-968437773d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.222426 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222337    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home" pod "a89dd4ba-b869-4902-9cd3-968437773d1f" (UID: "a89dd4ba-b869-4902-9cd3-968437773d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.222495 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222443    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir" pod "a89dd4ba-b869-4902-9cd3-968437773d1f" (UID: "a89dd4ba-b869-4902-9cd3-968437773d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.222656 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222637    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.222721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222663    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.222721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.222679    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.224360 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.224337    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs" pod "a89dd4ba-b869-4902-9cd3-968437773d1f" (UID: "a89dd4ba-b869-4902-9cd3-968437773d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:24:50.224442 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.224380    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm" pod "a89dd4ba-b869-4902-9cd3-968437773d1f" (UID: "a89dd4ba-b869-4902-9cd3-968437773d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.224501 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.224470    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w" pod "a89dd4ba-b869-4902-9cd3-968437773d1f" (UID: "a89dd4ba-b869-4902-9cd3-968437773d1f"). InnerVolumeSpecName "kube-api-access-p9s8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:50.272519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.272490    5515 generic.go:356] "Generic (PLEG): container finished" podID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerID="26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6" exitCode=0
Nov 03 04:24:50.272667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.272539    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"
Nov 03 04:24:50.272667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.272587    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" event={"ID":"d8627883-0bb5-4416-a42a-f0c5c9ef4252","Type":"ContainerDied","Data":"26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6"}
Nov 03 04:24:50.272667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.272637    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc" event={"ID":"d8627883-0bb5-4416-a42a-f0c5c9ef4252","Type":"ContainerDied","Data":"7e2743fb6ed781f199a15221e7e4d9d0e0bb90560a7740a9df449ab62d624725"}
Nov 03 04:24:50.272667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.272659    5515 scope.go:122] "RemoveContainer" containerID="26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6"
Nov 03 04:24:50.274141 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.274119    5515 generic.go:356] "Generic (PLEG): container finished" podID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerID="292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224" exitCode=0
Nov 03 04:24:50.274237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.274157    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" event={"ID":"a89dd4ba-b869-4902-9cd3-968437773d1f","Type":"ContainerDied","Data":"292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224"}
Nov 03 04:24:50.274237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.274167    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"
Nov 03 04:24:50.274237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.274180    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms" event={"ID":"a89dd4ba-b869-4902-9cd3-968437773d1f","Type":"ContainerDied","Data":"0d6e22763a7b00f9237d6962f21ddc9fd8770663f085551d6b6f5c6ae25f5085"}
Nov 03 04:24:50.281626 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.281527    5515 scope.go:122] "RemoveContainer" containerID="26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6"
Nov 03 04:24:50.281864 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:50.281836    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6\": container with ID starting with 26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6 not found: ID does not exist" containerID="26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6"
Nov 03 04:24:50.281968 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.281876    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6"} err="failed to get container status \"26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6\": rpc error: code = NotFound desc = could not find container \"26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6\": container with ID starting with 26e19dbe381a6daa7be1086d0734d2abdc866257dc375cbda35930b5299c34a6 not found: ID does not exist"
Nov 03 04:24:50.281968 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.281897    5515 scope.go:122] "RemoveContainer" containerID="292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224"
Nov 03 04:24:50.289583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.289566    5515 scope.go:122] "RemoveContainer" containerID="292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224"
Nov 03 04:24:50.289869 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:50.289845    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224\": container with ID starting with 292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224 not found: ID does not exist" containerID="292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224"
Nov 03 04:24:50.289927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.289876    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224"} err="failed to get container status \"292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224\": rpc error: code = NotFound desc = could not find container \"292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224\": container with ID starting with 292fa6d2c1211e0797ed18c134646d2abfad351bb0180a82a201587e105c3224 not found: ID does not exist"
Nov 03 04:24:50.323691 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323650    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir\") pod \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") "
Nov 03 04:24:50.323879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323700    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8\" (UniqueName: \"kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8\") pod \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") "
Nov 03 04:24:50.323879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323763    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home\") pod \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") "
Nov 03 04:24:50.323879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323828    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm\") pod \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") "
Nov 03 04:24:50.323879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323867    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache\") pod \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") "
Nov 03 04:24:50.324146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323897    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs\") pod \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\" (UID: \"d8627883-0bb5-4416-a42a-f0c5c9ef4252\") "
Nov 03 04:24:50.324146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.323928    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir" pod "d8627883-0bb5-4416-a42a-f0c5c9ef4252" (UID: "d8627883-0bb5-4416-a42a-f0c5c9ef4252"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.324146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324004    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home" pod "d8627883-0bb5-4416-a42a-f0c5c9ef4252" (UID: "d8627883-0bb5-4416-a42a-f0c5c9ef4252"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.324318 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324144    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache" pod "d8627883-0bb5-4416-a42a-f0c5c9ef4252" (UID: "d8627883-0bb5-4416-a42a-f0c5c9ef4252"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.324318 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324284    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.324318 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324305    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a89dd4ba-b869-4902-9cd3-968437773d1f-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.324494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324318    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.324494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324335    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s8w\" (UniqueName: \"kubernetes.io/projected/a89dd4ba-b869-4902-9cd3-968437773d1f-kube-api-access-p9s8w\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.324494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324351    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.324494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.324365    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a89dd4ba-b869-4902-9cd3-968437773d1f-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.326359 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.326113    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm" pod "d8627883-0bb5-4416-a42a-f0c5c9ef4252" (UID: "d8627883-0bb5-4416-a42a-f0c5c9ef4252"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:24:50.326359 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.326286    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs" pod "d8627883-0bb5-4416-a42a-f0c5c9ef4252" (UID: "d8627883-0bb5-4416-a42a-f0c5c9ef4252"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:24:50.326583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.326503    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8" pod "d8627883-0bb5-4416-a42a-f0c5c9ef4252" (UID: "d8627883-0bb5-4416-a42a-f0c5c9ef4252"). InnerVolumeSpecName "kube-api-access-45gg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:24:50.425737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.425650    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8627883-0bb5-4416-a42a-f0c5c9ef4252-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.425737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.425679    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8627883-0bb5-4416-a42a-f0c5c9ef4252-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.425737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.425689    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45gg8\" (UniqueName: \"kubernetes.io/projected/d8627883-0bb5-4416-a42a-f0c5c9ef4252-kube-api-access-45gg8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:24:50.539325 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.539294    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"]
Nov 03 04:24:50.543088 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.543064    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/premium-w0-0aa6b9-kserve-8d9c4748f-9lpms"]
Nov 03 04:24:50.694163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.694081    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"]
Nov 03 04:24:50.697941 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.697911    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0aa6b9/simulator-w0-0aa6b9-kserve-54865f5c6-cbxbc"]
Nov 03 04:24:50.893047 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.893015    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" path="/var/lib/kubelet/pods/a89dd4ba-b869-4902-9cd3-968437773d1f/volumes"
Nov 03 04:24:50.893440 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:50.893402    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" path="/var/lib/kubelet/pods/d8627883-0bb5-4416-a42a-f0c5c9ef4252/volumes"
Nov 03 04:24:50.983505 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:24:50.983416    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:24:52.127835 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:52.127798    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:52.928273 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:52.928236    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerName="payload-processing" containerID="cri-o://dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b" gracePeriod=25
Nov 03 04:24:53.037189 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:53.037144    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerName="payload-pre-processing" containerID="cri-o://c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e" gracePeriod=25
Nov 03 04:24:53.098790 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:53.098747    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:56.260740 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:56.260701    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:24:56.675630 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:56.675594    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:24:56.675833 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:56.675599    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:24:56.970144 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:56.970049    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:59.581066 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:59.581038    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:24:59.788853 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:24:59.788811    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:00.107091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:00.107062    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:01.201828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:01.201786    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:01.341758 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:01.341732    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:01.929940 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:01.929899    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:05.046076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:05.046039    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:06.011178 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:06.011135    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"]
Nov 03 04:25:06.409048 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:06.409010    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:06.551269 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:06.551238    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:06.676494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:06.676398    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:06.676675 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:06.676498    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:06.929569 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:06.929477    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:09.982049 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:09.982018    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:10.127432 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:10.127396    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:11.012485 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:11.012416    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerName="payload-pre-processing" containerID="cri-o://2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee" gracePeriod=25
Nov 03 04:25:11.299833 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:11.299752    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:11.496152 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:11.496123    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:12.471109 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.471075    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"]
Nov 03 04:25:12.471583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.471280    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" podUID="0c8cd813-2c64-4ce0-876b-2aabb23af795" containerName="maas-api" containerID="cri-o://6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571" gracePeriod=30
Nov 03 04:25:12.566575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.566529    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:12.731111 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.731050    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 04:25:12.760345 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.760317    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\") pod \"0c8cd813-2c64-4ce0-876b-2aabb23af795\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") "
Nov 03 04:25:12.760525 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.760352    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca\") pod \"0c8cd813-2c64-4ce0-876b-2aabb23af795\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") "
Nov 03 04:25:12.760525 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.760412    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb\" (UniqueName: \"kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb\") pod \"0c8cd813-2c64-4ce0-876b-2aabb23af795\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") "
Nov 03 04:25:12.760525 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.760457    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs\") pod \"0c8cd813-2c64-4ce0-876b-2aabb23af795\" (UID: \"0c8cd813-2c64-4ce0-876b-2aabb23af795\") "
Nov 03 04:25:12.760823 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.760798    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca" pod "0c8cd813-2c64-4ce0-876b-2aabb23af795" (UID: "0c8cd813-2c64-4ce0-876b-2aabb23af795"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:12.760938 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.760922    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8cd813-2c64-4ce0-876b-2aabb23af795-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:12.762711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.762686    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb" pod "0c8cd813-2c64-4ce0-876b-2aabb23af795" (UID: "0c8cd813-2c64-4ce0-876b-2aabb23af795"). InnerVolumeSpecName "kube-api-access-skvnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:12.762711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.762692    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls" pod "0c8cd813-2c64-4ce0-876b-2aabb23af795" (UID: "0c8cd813-2c64-4ce0-876b-2aabb23af795"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:25:12.763170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.762725    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs" pod "0c8cd813-2c64-4ce0-876b-2aabb23af795" (UID: "0c8cd813-2c64-4ce0-876b-2aabb23af795"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:25:12.861576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.861524    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skvnb\" (UniqueName: \"kubernetes.io/projected/0c8cd813-2c64-4ce0-876b-2aabb23af795-kube-api-access-skvnb\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:12.861576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.861571    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:12.861576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:12.861582    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8cd813-2c64-4ce0-876b-2aabb23af795-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:13.163673 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.163648    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 04:25:13.269668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.269637    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume\") pod \"e65523bf-94e1-48f5-8fe1-37baaf65636c\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") "
Nov 03 04:25:13.269857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.269762    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8\" (UniqueName: \"kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8\") pod \"e65523bf-94e1-48f5-8fe1-37baaf65636c\" (UID: \"e65523bf-94e1-48f5-8fe1-37baaf65636c\") "
Nov 03 04:25:13.270092 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.270064    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume" pod "e65523bf-94e1-48f5-8fe1-37baaf65636c" (UID: "e65523bf-94e1-48f5-8fe1-37baaf65636c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:13.272370 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.272343    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8" pod "e65523bf-94e1-48f5-8fe1-37baaf65636c" (UID: "e65523bf-94e1-48f5-8fe1-37baaf65636c"). InnerVolumeSpecName "kube-api-access-mzch8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:13.285512 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.285483    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 04:25:13.371006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.370969    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x\" (UniqueName: \"kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x\") pod \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") "
Nov 03 04:25:13.371205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.371032    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume\") pod \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\" (UID: \"a0dc0268-3d46-40d8-8e4b-d5299cd9d074\") "
Nov 03 04:25:13.371393 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.371368    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume" pod "a0dc0268-3d46-40d8-8e4b-d5299cd9d074" (UID: "a0dc0268-3d46-40d8-8e4b-d5299cd9d074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:13.371465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.371435    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzch8\" (UniqueName: \"kubernetes.io/projected/e65523bf-94e1-48f5-8fe1-37baaf65636c-kube-api-access-mzch8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:13.371465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.371457    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65523bf-94e1-48f5-8fe1-37baaf65636c-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:13.373228 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.373207    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x" pod "a0dc0268-3d46-40d8-8e4b-d5299cd9d074" (UID: "a0dc0268-3d46-40d8-8e4b-d5299cd9d074"). InnerVolumeSpecName "kube-api-access-wrh7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:13.404001 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.403967    5515 generic.go:356] "Generic (PLEG): container finished" podID="0c8cd813-2c64-4ce0-876b-2aabb23af795" containerID="6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571" exitCode=0
Nov 03 04:25:13.404234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.404124    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"
Nov 03 04:25:13.404710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.404645    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" event={"ID":"0c8cd813-2c64-4ce0-876b-2aabb23af795","Type":"ContainerDied","Data":"6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571"}
Nov 03 04:25:13.406687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.405705    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj" event={"ID":"0c8cd813-2c64-4ce0-876b-2aabb23af795","Type":"ContainerDied","Data":"8619b00c3fa0959583a87acbcb6ecffe14ee8d4ce9513090eeca56040ed63205"}
Nov 03 04:25:13.406687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.405756    5515 scope.go:122] "RemoveContainer" containerID="6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571"
Nov 03 04:25:13.409023 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.408992    5515 generic.go:356] "Generic (PLEG): container finished" podID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerID="dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b" exitCode=0
Nov 03 04:25:13.409994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.409269    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" event={"ID":"e65523bf-94e1-48f5-8fe1-37baaf65636c","Type":"ContainerDied","Data":"dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b"}
Nov 03 04:25:13.409994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.409317    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6" event={"ID":"e65523bf-94e1-48f5-8fe1-37baaf65636c","Type":"ContainerDied","Data":"e4880df36093c5095f0ad22ee1dcfba1f83db175cf37e6cb483468d8d70fee97"}
Nov 03 04:25:13.409994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.409398    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"
Nov 03 04:25:13.414612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.414166    5515 generic.go:356] "Generic (PLEG): container finished" podID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerID="c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e" exitCode=0
Nov 03 04:25:13.414612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.414243    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" event={"ID":"a0dc0268-3d46-40d8-8e4b-d5299cd9d074","Type":"ContainerDied","Data":"c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e"}
Nov 03 04:25:13.414612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.414281    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4" event={"ID":"a0dc0268-3d46-40d8-8e4b-d5299cd9d074","Type":"ContainerDied","Data":"22075c80af8f87375ccda9e3397d5ce1cb138eb88a05ceca9e520baf47aa6e99"}
Nov 03 04:25:13.414612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.414358    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"
Nov 03 04:25:13.424672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.424531    5515 scope.go:122] "RemoveContainer" containerID="6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571"
Nov 03 04:25:13.425396 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:13.424956    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571\": container with ID starting with 6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571 not found: ID does not exist" containerID="6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571"
Nov 03 04:25:13.425396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.424993    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571"} err="failed to get container status \"6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571\": rpc error: code = NotFound desc = could not find container \"6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571\": container with ID starting with 6800d9874ac08e22668a6f4310b266542dbadbb8b4c275ca6298ac7127980571 not found: ID does not exist"
Nov 03 04:25:13.425396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.425013    5515 scope.go:122] "RemoveContainer" containerID="dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b"
Nov 03 04:25:13.439988 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.439968    5515 scope.go:122] "RemoveContainer" containerID="dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b"
Nov 03 04:25:13.440330 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:13.440311    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b\": container with ID starting with dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b not found: ID does not exist" containerID="dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b"
Nov 03 04:25:13.440413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.440340    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b"} err="failed to get container status \"dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b\": rpc error: code = NotFound desc = could not find container \"dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b\": container with ID starting with dc83c86b26f1652ad519ed434348ca85dddd32e3f63a91b5cc9160f7b020e21b not found: ID does not exist"
Nov 03 04:25:13.440413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.440357    5515 scope.go:122] "RemoveContainer" containerID="c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e"
Nov 03 04:25:13.444674 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.444637    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:13.450690 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.450675    5515 scope.go:122] "RemoveContainer" containerID="c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e"
Nov 03 04:25:13.450940 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:13.450924    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e\": container with ID starting with c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e not found: ID does not exist" containerID="c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e"
Nov 03 04:25:13.450989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.450948    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e"} err="failed to get container status \"c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e\": rpc error: code = NotFound desc = could not find container \"c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e\": container with ID starting with c9b90a27ab025b1da37d679323e2986e02b54f9ca1c49d31102b58920d4b8b2e not found: ID does not exist"
Nov 03 04:25:13.472540 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.472516    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrh7x\" (UniqueName: \"kubernetes.io/projected/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-kube-api-access-wrh7x\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:13.472899 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.472542    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dc0268-3d46-40d8-8e4b-d5299cd9d074-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:13.571112 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.571075    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"]
Nov 03 04:25:13.574787 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.574766    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-0aa6b9-86b9ff4b95-qg7t4"]
Nov 03 04:25:13.684516 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.684436    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"]
Nov 03 04:25:13.688584 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.688557    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0aa6b9-86c9c8ddbf-sclt6"]
Nov 03 04:25:13.799208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.799172    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"]
Nov 03 04:25:13.803343 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:13.803319    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0aa6b9-7d66d4d599-lw6gj"]
Nov 03 04:25:14.095123 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:14.095045    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:14.322463 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:14.322432    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:14.877398 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:14.877343    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:14.892918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:14.892892    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8cd813-2c64-4ce0-876b-2aabb23af795" path="/var/lib/kubelet/pods/0c8cd813-2c64-4ce0-876b-2aabb23af795/volumes"
Nov 03 04:25:14.893279 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:14.893266    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" path="/var/lib/kubelet/pods/a0dc0268-3d46-40d8-8e4b-d5299cd9d074/volumes"
Nov 03 04:25:14.893612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:14.893600    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" path="/var/lib/kubelet/pods/e65523bf-94e1-48f5-8fe1-37baaf65636c/volumes"
Nov 03 04:25:15.015879 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:15.015849    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:16.469837 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:16.469804    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:16.770475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:16.770399    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:16.994900 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:16.994852    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"]
Nov 03 04:25:16.995109 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:16.995080    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" containerID="cri-o://0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273" gracePeriod=30
Nov 03 04:25:17.008877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:17.008848    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:18.164897 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:18.164861    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:25:19.095399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:19.095356    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:19.658045 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:19.658012    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:20.650928 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:20.650893    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:20.915277 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:20.915193    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:20.986567 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:20.986478    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:25:21.058890 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:21.058856    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:22.234925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.234901    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 04:25:22.367475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367445    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367516    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367583    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367617    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367646    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367712    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367746    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367803    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.367908 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367835    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert\") pod \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\" (UID: \"f8623e33-0051-4ac5-b370-53fc00aa6fe6\") "
Nov 03 04:25:22.368133 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367878    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:22.368133 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367972    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:22.368133 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.367998    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:22.368314 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.368149    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:22.368314 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.368244    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.368415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.368318    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:22.368415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.368334    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.368507 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.368444    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.370189 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.370142    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "kube-api-access-mnnz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:22.370326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.370215    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:22.370806 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.370762    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:22.370980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.370948    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo" pod "f8623e33-0051-4ac5-b370-53fc00aa6fe6" (UID: "f8623e33-0051-4ac5-b370-53fc00aa6fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:25:22.434072 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.434039    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:22.463026 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.462992    5515 generic.go:356] "Generic (PLEG): container finished" podID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerID="0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273" exitCode=0
Nov 03 04:25:22.463192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.463051    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"
Nov 03 04:25:22.463192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.463078    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" event={"ID":"f8623e33-0051-4ac5-b370-53fc00aa6fe6","Type":"ContainerDied","Data":"0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273"}
Nov 03 04:25:22.463192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.463135    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" event={"ID":"f8623e33-0051-4ac5-b370-53fc00aa6fe6","Type":"ContainerDied","Data":"c1e919bca69e4335953d17e6f47f3c5484b1324428b1170c7469292291729adb"}
Nov 03 04:25:22.463192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.463162    5515 scope.go:122] "RemoveContainer" containerID="0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273"
Nov 03 04:25:22.469205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.469180    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnnz4\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-kube-api-access-mnnz4\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.469205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.469206    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.469379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.469217    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.469379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.469226    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8623e33-0051-4ac5-b370-53fc00aa6fe6-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.469379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.469235    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.469379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.469243    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8623e33-0051-4ac5-b370-53fc00aa6fe6-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:22.472745 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.472724    5515 scope.go:122] "RemoveContainer" containerID="968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"
Nov 03 04:25:22.492501 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.492479    5515 scope.go:122] "RemoveContainer" containerID="0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273"
Nov 03 04:25:22.492885 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:22.492865    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273\": container with ID starting with 0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273 not found: ID does not exist" containerID="0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273"
Nov 03 04:25:22.492966 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.492898    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273"} err="failed to get container status \"0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273\": rpc error: code = NotFound desc = could not find container \"0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273\": container with ID starting with 0156c36d1da80277f9e5504d60a96add4931859de6761890df1ada4e67503273 not found: ID does not exist"
Nov 03 04:25:22.492966 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.492919    5515 scope.go:122] "RemoveContainer" containerID="968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"
Nov 03 04:25:22.493220 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:22.493203    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da\": container with ID starting with 968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da not found: ID does not exist" containerID="968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"
Nov 03 04:25:22.493286 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.493225    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da"} err="failed to get container status \"968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da\": rpc error: code = NotFound desc = could not find container \"968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da\": container with ID starting with 968baf3b5725385ce663b146e3bc8a57d02144d29afb188b460699522f9fe3da not found: ID does not exist"
Nov 03 04:25:22.630307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.630227    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"]
Nov 03 04:25:22.634490 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.634461    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882"]
Nov 03 04:25:22.893621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.893533    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" path="/var/lib/kubelet/pods/f8623e33-0051-4ac5-b370-53fc00aa6fe6/volumes"
Nov 03 04:25:22.925788 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.925753    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"]
Nov 03 04:25:22.926030 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.926006    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy" containerID="cri-o://538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43" gracePeriod=30
Nov 03 04:25:22.959646 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:22.959622    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:23.216914 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:23.216826    5515 patch_prober.go:29] interesting pod/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:25:23.216914 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:23.216883    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0aa6b9-openshift-default-78766ccf59-z6882" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:25:23.795809 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:23.795771    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:24.999903 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:24.999876    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:28.175293 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.175260    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 04:25:28.331020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.330952    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331015    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331033    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331056    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331076    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331092    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331115    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331145    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331242    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:28.331644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331253    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket\") pod \"517b403f-ed14-4aa2-8aee-3943ac901a12\" (UID: \"517b403f-ed14-4aa2-8aee-3943ac901a12\") "
Nov 03 04:25:28.331644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331471    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:28.331644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331529    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:28.331779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331663    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:28.331779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331677    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:28.331863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331808    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.331863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331829    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.331863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331845    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517b403f-ed14-4aa2-8aee-3943ac901a12-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.331863 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331855    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.331996 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.331865    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.333444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.333413    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "kube-api-access-fqtkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:28.333767 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.333745    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:25:28.333976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.333959    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:25:28.334027 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.333991    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token" pod "517b403f-ed14-4aa2-8aee-3943ac901a12" (UID: "517b403f-ed14-4aa2-8aee-3943ac901a12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:28.432721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.432692    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.432721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.432717    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqtkl\" (UniqueName: \"kubernetes.io/projected/517b403f-ed14-4aa2-8aee-3943ac901a12-kube-api-access-fqtkl\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.432721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.432727    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.432956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.432736    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517b403f-ed14-4aa2-8aee-3943ac901a12-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:28.494353 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.494324    5515 generic.go:356] "Generic (PLEG): container finished" podID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerID="538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43" exitCode=0
Nov 03 04:25:28.494487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.494371    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"
Nov 03 04:25:28.494487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.494415    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" event={"ID":"517b403f-ed14-4aa2-8aee-3943ac901a12","Type":"ContainerDied","Data":"538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43"}
Nov 03 04:25:28.494487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.494466    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr" event={"ID":"517b403f-ed14-4aa2-8aee-3943ac901a12","Type":"ContainerDied","Data":"caa9abf2c973b782dd23e516b9634ccaf4e6ff185b735cd59a8dd1c75d8cda74"}
Nov 03 04:25:28.494685 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.494488    5515 scope.go:122] "RemoveContainer" containerID="538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43"
Nov 03 04:25:28.504706 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.504655    5515 scope.go:122] "RemoveContainer" containerID="72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"
Nov 03 04:25:28.512749 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.512729    5515 scope.go:122] "RemoveContainer" containerID="538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43"
Nov 03 04:25:28.513024 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:28.513005    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43\": container with ID starting with 538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43 not found: ID does not exist" containerID="538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43"
Nov 03 04:25:28.513070 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.513035    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43"} err="failed to get container status \"538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43\": rpc error: code = NotFound desc = could not find container \"538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43\": container with ID starting with 538c77cb95a40e72e3d5974af09dac6b3754a43d3f92352fbed4fa0978dbba43 not found: ID does not exist"
Nov 03 04:25:28.513070 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.513054    5515 scope.go:122] "RemoveContainer" containerID="72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"
Nov 03 04:25:28.513291 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:28.513266    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82\": container with ID starting with 72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82 not found: ID does not exist" containerID="72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"
Nov 03 04:25:28.513384 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.513296    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82"} err="failed to get container status \"72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82\": rpc error: code = NotFound desc = could not find container \"72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82\": container with ID starting with 72f800b2d962fa7c342a054ec8e4acdf141d0b096f817e2da4490f4a63ab3c82 not found: ID does not exist"
Nov 03 04:25:28.661315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.661287    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"]
Nov 03 04:25:28.673719 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.668482    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-66d902-openshift-default-86cf8f74b5-c7frr"]
Nov 03 04:25:28.892890 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:28.892859    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" path="/var/lib/kubelet/pods/517b403f-ed14-4aa2-8aee-3943ac901a12/volumes"
Nov 03 04:25:29.096167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:29.096084    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:29.522100 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:29.522067    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"]
Nov 03 04:25:29.630571 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:29.630512    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"]
Nov 03 04:25:29.878020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:29.877989    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:30.022885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:30.022850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:30.162056 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:30.161967    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:30.682575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:30.682521    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:31.278139 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.278081    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 04:25:31.360823 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.360792    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume\") pod \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") "
Nov 03 04:25:31.361002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.360896    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n\" (UniqueName: \"kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n\") pod \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\" (UID: \"aa654b1a-05f5-4154-b259-bc74b75cc1c9\") "
Nov 03 04:25:31.361194 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.361172    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume" pod "aa654b1a-05f5-4154-b259-bc74b75cc1c9" (UID: "aa654b1a-05f5-4154-b259-bc74b75cc1c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:31.363768 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.363743    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n" pod "aa654b1a-05f5-4154-b259-bc74b75cc1c9" (UID: "aa654b1a-05f5-4154-b259-bc74b75cc1c9"). InnerVolumeSpecName "kube-api-access-j2b7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:31.461710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.461679    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2b7n\" (UniqueName: \"kubernetes.io/projected/aa654b1a-05f5-4154-b259-bc74b75cc1c9-kube-api-access-j2b7n\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:31.461710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.461706    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa654b1a-05f5-4154-b259-bc74b75cc1c9-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:31.509294 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.509260    5515 generic.go:356] "Generic (PLEG): container finished" podID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerID="2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee" exitCode=0
Nov 03 04:25:31.509438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.509304    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"
Nov 03 04:25:31.509438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.509349    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" event={"ID":"aa654b1a-05f5-4154-b259-bc74b75cc1c9","Type":"ContainerDied","Data":"2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee"}
Nov 03 04:25:31.509438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.509401    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd" event={"ID":"aa654b1a-05f5-4154-b259-bc74b75cc1c9","Type":"ContainerDied","Data":"bf12cdf1b549e28726b6c8b085fc21fd2102cdde717f8599b7a42799ac688aca"}
Nov 03 04:25:31.509438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.509432    5515 scope.go:122] "RemoveContainer" containerID="2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee"
Nov 03 04:25:31.518281 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.518265    5515 scope.go:122] "RemoveContainer" containerID="2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee"
Nov 03 04:25:31.518573 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:31.518535    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee\": container with ID starting with 2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee not found: ID does not exist" containerID="2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee"
Nov 03 04:25:31.518648 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.518580    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee"} err="failed to get container status \"2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee\": rpc error: code = NotFound desc = could not find container \"2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee\": container with ID starting with 2805d32b07ebd4f0d005948d8b67c360a72a484fc8600fb0a256209ce2e9cfee not found: ID does not exist"
Nov 03 04:25:31.675255 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.675220    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"]
Nov 03 04:25:31.681090 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:31.681056    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-66d902-7cf479d6d7-qlvsd"]
Nov 03 04:25:32.892208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:32.892168    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" path="/var/lib/kubelet/pods/aa654b1a-05f5-4154-b259-bc74b75cc1c9/volumes"
Nov 03 04:25:34.522600 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:34.522533    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerName="payload-processing" containerID="cri-o://4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf" gracePeriod=25
Nov 03 04:25:34.631838 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:34.631790    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerName="payload-pre-processing" containerID="cri-o://87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c" gracePeriod=25
Nov 03 04:25:34.750930 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:34.750892    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:35.826636 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:35.826598    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:35.975844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:35.975810    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:36.519037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:36.519003    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:36.781757 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:36.781668    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:38.080342 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:38.080306    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:38.656458 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:38.656417    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:39.173508 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:39.173479    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:41.621585 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:41.621532    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:41.621999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:41.621602    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:41.890644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:41.890535    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:42.926955 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:42.926922    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:25:44.296449 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:44.296413    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:46.932819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:46.932784    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:50.983240 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:50.983205    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:25:51.511820 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:51.511792    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:51.621164 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:51.621129    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:51.621335 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:51.621238    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:25:51.814266 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:51.814171    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:54.779470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.779412    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 04:25:54.879595 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.879571    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 04:25:54.881680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.881658    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b\" (UniqueName: \"kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b\") pod \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") "
Nov 03 04:25:54.881824 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.881715    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume\") pod \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\" (UID: \"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9\") "
Nov 03 04:25:54.882113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.882094    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume" pod "0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" (UID: "0c1e6ce0-a826-4a6c-b6ac-75396e9825f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:54.883878 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.883854    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b" pod "0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" (UID: "0c1e6ce0-a826-4a6c-b6ac-75396e9825f9"). InnerVolumeSpecName "kube-api-access-qpj9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:54.983183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.983156    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr\" (UniqueName: \"kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr\") pod \"3a3de473-ca1b-4cdd-9696-d291633413b2\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") "
Nov 03 04:25:54.983364 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.983321    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume\") pod \"3a3de473-ca1b-4cdd-9696-d291633413b2\" (UID: \"3a3de473-ca1b-4cdd-9696-d291633413b2\") "
Nov 03 04:25:54.983590 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.983540    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpj9b\" (UniqueName: \"kubernetes.io/projected/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-kube-api-access-qpj9b\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:54.983705 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.983595    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:54.983748 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.983705    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume" pod "3a3de473-ca1b-4cdd-9696-d291633413b2" (UID: "3a3de473-ca1b-4cdd-9696-d291633413b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:25:54.985526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:54.985510    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr" pod "3a3de473-ca1b-4cdd-9696-d291633413b2" (UID: "3a3de473-ca1b-4cdd-9696-d291633413b2"). InnerVolumeSpecName "kube-api-access-m42hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:25:55.084963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.084874    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3de473-ca1b-4cdd-9696-d291633413b2-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:55.084963 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.084903    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42hr\" (UniqueName: \"kubernetes.io/projected/3a3de473-ca1b-4cdd-9696-d291633413b2-kube-api-access-m42hr\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:25:55.625428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.625390    5515 generic.go:356] "Generic (PLEG): container finished" podID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerID="87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c" exitCode=0
Nov 03 04:25:55.625657 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.625437    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"
Nov 03 04:25:55.625657 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.625476    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" event={"ID":"3a3de473-ca1b-4cdd-9696-d291633413b2","Type":"ContainerDied","Data":"87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c"}
Nov 03 04:25:55.625657 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.625527    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c" event={"ID":"3a3de473-ca1b-4cdd-9696-d291633413b2","Type":"ContainerDied","Data":"261ea5a7007da04d124a7c8e8c71cf644ace96fcf72349569b6c48fa9d9fee24"}
Nov 03 04:25:55.625657 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.625572    5515 scope.go:122] "RemoveContainer" containerID="87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c"
Nov 03 04:25:55.626948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.626926    5515 generic.go:356] "Generic (PLEG): container finished" podID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerID="4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf" exitCode=0
Nov 03 04:25:55.627121 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.626981    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"
Nov 03 04:25:55.627121 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.627003    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" event={"ID":"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9","Type":"ContainerDied","Data":"4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf"}
Nov 03 04:25:55.627121 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.627044    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx" event={"ID":"0c1e6ce0-a826-4a6c-b6ac-75396e9825f9","Type":"ContainerDied","Data":"752a4cb1d80ecb85a77315d9e4044df9d000affec1e872a6475b5e29ee85e4d4"}
Nov 03 04:25:55.635126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.635111    5515 scope.go:122] "RemoveContainer" containerID="87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c"
Nov 03 04:25:55.635361 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:55.635344    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c\": container with ID starting with 87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c not found: ID does not exist" containerID="87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c"
Nov 03 04:25:55.635436 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.635367    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c"} err="failed to get container status \"87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c\": rpc error: code = NotFound desc = could not find container \"87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c\": container with ID starting with 87a26183971e764c3bb80f0856cc7a3e1c1534d7e3f7500586509bdf489e172c not found: ID does not exist"
Nov 03 04:25:55.635436 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.635381    5515 scope.go:122] "RemoveContainer" containerID="4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf"
Nov 03 04:25:55.643787 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.643625    5515 scope.go:122] "RemoveContainer" containerID="4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf"
Nov 03 04:25:55.643908 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:25:55.643891    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf\": container with ID starting with 4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf not found: ID does not exist" containerID="4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf"
Nov 03 04:25:55.643949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.643915    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf"} err="failed to get container status \"4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf\": rpc error: code = NotFound desc = could not find container \"4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf\": container with ID starting with 4cc831422ecfca50b7dbc12488fed8582382b528fe80745b48496f4f7c06facf not found: ID does not exist"
Nov 03 04:25:55.790868 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.790789    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"]
Nov 03 04:25:55.841597 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.841564    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3d8dc5-7cc8b55c5-rg52c"]
Nov 03 04:25:55.951869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.951837    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"]
Nov 03 04:25:55.962619 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:55.956395    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3d8dc5-5c55b56569-rrfcx"]
Nov 03 04:25:56.179648 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:56.179617    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:25:56.781701 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:56.781644    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:56.896371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:56.896332    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" path="/var/lib/kubelet/pods/0c1e6ce0-a826-4a6c-b6ac-75396e9825f9/volumes"
Nov 03 04:25:56.896979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:56.896956    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" path="/var/lib/kubelet/pods/3a3de473-ca1b-4cdd-9696-d291633413b2/volumes"
Nov 03 04:25:58.337196 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:58.337160    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:58.897434 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:58.897396    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"]
Nov 03 04:25:59.262939 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:59.262850    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:25:59.408867 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:25:59.408829    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:02.041593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:02.041538    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:03.897862 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:03.897797    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerName="payload-processing" containerID="cri-o://5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff" gracePeriod=25
Nov 03 04:26:04.449349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:04.449312    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:08.559758 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:08.559715    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:26:10.193366 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:10.193333    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:11.410419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:11.410348    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:15.917292 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:15.917259    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:16.072543 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:16.072513    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:17.130807 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:17.130778    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:17.400979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:17.400899    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:17.550983 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:17.550950    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:18.467858 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:18.467825    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:26:18.560043 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:18.559999    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:26:20.983457 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:26:20.983419    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:26:21.357661 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:21.357634    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:21.504455 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:21.504424    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:22.322796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:22.322757    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:24.148501 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.148478    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 04:26:24.163886 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.163837    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume\") pod \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") "
Nov 03 04:26:24.164010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.163884    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt\" (UniqueName: \"kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt\") pod \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\" (UID: \"266b0b26-9296-4a87-aa4e-a7ef5d3104ab\") "
Nov 03 04:26:24.164180 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.164159    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume" pod "266b0b26-9296-4a87-aa4e-a7ef5d3104ab" (UID: "266b0b26-9296-4a87-aa4e-a7ef5d3104ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:26:24.166038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.166017    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt" pod "266b0b26-9296-4a87-aa4e-a7ef5d3104ab" (UID: "266b0b26-9296-4a87-aa4e-a7ef5d3104ab"). InnerVolumeSpecName "kube-api-access-fpfqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:26:24.265306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.265259    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:24.265306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.265301    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpfqt\" (UniqueName: \"kubernetes.io/projected/266b0b26-9296-4a87-aa4e-a7ef5d3104ab-kube-api-access-fpfqt\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:24.697625 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.697526    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:24.763649 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.763608    5515 generic.go:356] "Generic (PLEG): container finished" podID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerID="5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff" exitCode=0
Nov 03 04:26:24.763825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.763663    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"
Nov 03 04:26:24.763825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.763684    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" event={"ID":"266b0b26-9296-4a87-aa4e-a7ef5d3104ab","Type":"ContainerDied","Data":"5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff"}
Nov 03 04:26:24.763825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.763764    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4" event={"ID":"266b0b26-9296-4a87-aa4e-a7ef5d3104ab","Type":"ContainerDied","Data":"5ce52db27bc128b68cd80282d8780e60c4b0ee342c4ba565e896ab975744e1aa"}
Nov 03 04:26:24.763825 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.763787    5515 scope.go:122] "RemoveContainer" containerID="5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff"
Nov 03 04:26:24.772870 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.772853    5515 scope.go:122] "RemoveContainer" containerID="5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff"
Nov 03 04:26:24.773159 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:26:24.773136    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff\": container with ID starting with 5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff not found: ID does not exist" containerID="5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff"
Nov 03 04:26:24.773218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.773167    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff"} err="failed to get container status \"5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff\": rpc error: code = NotFound desc = could not find container \"5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff\": container with ID starting with 5569891bd5e87e0dcd153e03a9487bcb622cda638522a6eb34fcb1c79df57eff not found: ID does not exist"
Nov 03 04:26:24.844205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.844179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:24.931648 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.931609    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"]
Nov 03 04:26:24.982415 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:24.982330    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-de27f5-6b8948f649-48bb4"]
Nov 03 04:26:26.299873 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:26.299840    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:26.826229 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:26.826198    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"]
Nov 03 04:26:26.826482 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:26.826457    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy" containerID="cri-o://c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101" gracePeriod=30
Nov 03 04:26:26.832577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:26.832509    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:26.892186 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:26.892160    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" path="/var/lib/kubelet/pods/266b0b26-9296-4a87-aa4e-a7ef5d3104ab/volumes"
Nov 03 04:26:28.372868 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:28.372836    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:28.530468 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:28.530442    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:28.983984 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:28.983956    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-m5qlx_5c3cc519-a61c-4b59-905e-6c06aa56da13/manager/0.log"
Nov 03 04:26:29.196602 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:29.196573    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sj294_0f8f7050-3fb9-4229-b35a-b5d831cdb957/kuadrant-console-plugin/0.log"
Nov 03 04:26:29.883948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:29.883917    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhs8n_83d03a51-825c-477b-9fcc-e5af9421d387/limitador/0.log"
Nov 03 04:26:31.169172 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:31.169141    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d99f9cf84-h9p42_93ba1b83-66ba-4138-ae86-98b9dd588eed/manager/0.log"
Nov 03 04:26:31.539740 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:31.539666    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7tdsv_108ea924-255e-45c4-85a3-deaa09ddb1f0/manager/0.log"
Nov 03 04:26:32.075216 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.075192    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 04:26:32.139833 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.139811    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.139931 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.139848    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.139931 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.139886    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.139931 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.139915    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.140085 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.139978    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.140085 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140030    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.140198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140105    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.140198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140135    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.140198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140175    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo\") pod \"cc82122b-3adb-4448-897a-4165d1c711b1\" (UID: \"cc82122b-3adb-4448-897a-4165d1c711b1\") "
Nov 03 04:26:32.140365 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140159    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:26:32.140365 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140285    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:26:32.140490 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140448    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:26:32.140696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140622    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:26:32.140696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140648    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:26:32.140696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140648    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.140696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140755    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.141119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.140770    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.142192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.142167    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "kube-api-access-9gzpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:26:32.142192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.142177    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:26:32.142444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.142376    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:26:32.142692 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.142672    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo" pod "cc82122b-3adb-4448-897a-4165d1c711b1" (UID: "cc82122b-3adb-4448-897a-4165d1c711b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:26:32.242109 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.242083    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc82122b-3adb-4448-897a-4165d1c711b1-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.242109 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.242105    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc82122b-3adb-4448-897a-4165d1c711b1-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.242465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.242114    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.242465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.242123    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gzpn\" (UniqueName: \"kubernetes.io/projected/cc82122b-3adb-4448-897a-4165d1c711b1-kube-api-access-9gzpn\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.242465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.242134    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.242465 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.242141    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc82122b-3adb-4448-897a-4165d1c711b1-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:26:32.803342 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.803308    5515 generic.go:356] "Generic (PLEG): container finished" podID="cc82122b-3adb-4448-897a-4165d1c711b1" containerID="c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101" exitCode=0
Nov 03 04:26:32.803592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.803350    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"
Nov 03 04:26:32.803592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.803372    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" event={"ID":"cc82122b-3adb-4448-897a-4165d1c711b1","Type":"ContainerDied","Data":"c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101"}
Nov 03 04:26:32.803592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.803431    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx" event={"ID":"cc82122b-3adb-4448-897a-4165d1c711b1","Type":"ContainerDied","Data":"b1d231e08b5aee7322dc37d7aecb550807c778bde72e0559c7ba92a7fb643a52"}
Nov 03 04:26:32.803592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.803459    5515 scope.go:122] "RemoveContainer" containerID="c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101"
Nov 03 04:26:32.813282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.813263    5515 scope.go:122] "RemoveContainer" containerID="c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101"
Nov 03 04:26:32.813497 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:26:32.813477    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101\": container with ID starting with c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101 not found: ID does not exist" containerID="c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101"
Nov 03 04:26:32.813574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:32.813505    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101"} err="failed to get container status \"c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101\": rpc error: code = NotFound desc = could not find container \"c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101\": container with ID starting with c88405bceefa2bc02279fdeb570e845b6f5d24856c23fb0a20ce261f336ed101 not found: ID does not exist"
Nov 03 04:26:33.022230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:33.022200    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"]
Nov 03 04:26:33.073093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:33.073022    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-de27f5-openshift-default-7bd5bf66c-fq9xx"]
Nov 03 04:26:33.599456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:33.599430    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-discovery-845d48d7b4-6k262_7ccb2946-a5ce-4a22-b5eb-301c6373c834/maas-discovery/0.log"
Nov 03 04:26:34.893163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:34.893134    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" path="/var/lib/kubelet/pods/cc82122b-3adb-4448-897a-4165d1c711b1/volumes"
Nov 03 04:26:39.388500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.388469    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:39.912278 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912240    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"]
Nov 03 04:26:39.912747 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912728    5515 cpu_manager.go:414] "containerMap: removing container" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912748    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912766    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912773    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912786    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912794    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912810    5515 cpu_manager.go:414] "containerMap: removing container" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912818    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912828    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912834    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912844    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:26:39.912856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912851    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912865    5515 cpu_manager.go:414] "containerMap: removing container" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912871    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912884    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912892    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912904    5515 cpu_manager.go:414] "containerMap: removing container" podUID="668a3983-44bd-4f91-954d-7c501263b172" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912911    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668a3983-44bd-4f91-954d-7c501263b172" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912925    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0c8cd813-2c64-4ce0-876b-2aabb23af795" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912931    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8cd813-2c64-4ce0-876b-2aabb23af795" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912940    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912948    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912961    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912968    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912983    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.912990    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913001    5515 cpu_manager.go:414] "containerMap: removing container" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913009    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913017    5515 cpu_manager.go:414] "containerMap: removing container" podUID="428d296d-b09c-4d16-8486-7f12791481bb" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913024    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428d296d-b09c-4d16-8486-7f12791481bb" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913035    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913042    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913051    5515 cpu_manager.go:414] "containerMap: removing container" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913059    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913069    5515 cpu_manager.go:414] "containerMap: removing container" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913075    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913088    5515 cpu_manager.go:414] "containerMap: removing container" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913096    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913106    5515 cpu_manager.go:414] "containerMap: removing container" podUID="39620b2e-b2d5-4162-8115-d40e81939478" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913113    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39620b2e-b2d5-4162-8115-d40e81939478" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913125    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913132    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913145    5515 cpu_manager.go:414] "containerMap: removing container" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913152    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913160    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f2361be4-77ae-456b-99f6-80f3c5aa0aed" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913167    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2361be4-77ae-456b-99f6-80f3c5aa0aed" containerName="maas-api"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913186    5515 cpu_manager.go:414] "containerMap: removing container" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913193    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913206    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913214    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913227    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913234    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913245    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913252    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913264    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8bf262dc-2456-41aa-a3d1-5bcd7126583d" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913271    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf262dc-2456-41aa-a3d1-5bcd7126583d" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913282    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913290    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913302    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913309    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913318    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913325    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerName="payload-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913335    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913343    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913352    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913359    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913369    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913376    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913390    5515 cpu_manager.go:414] "containerMap: removing container" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913397    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913408    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913414    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913425    5515 cpu_manager.go:414] "containerMap: removing container" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913432    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913444    5515 cpu_manager.go:414] "containerMap: removing container" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913451    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fedb24-5cd5-4edc-befe-376f05ea6231" containerName="cleanup"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913462    5515 cpu_manager.go:414] "containerMap: removing container" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913469    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913480    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913487    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913496    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913503    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerName="payload-pre-processing"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913518    5515 cpu_manager.go:414] "containerMap: removing container" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main"
Nov 03 04:26:39.913531 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913524    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913650    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dc0268-3d46-40d8-8e4b-d5299cd9d074" containerName="payload-pre-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913668    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="246d8290-3d5f-405a-96bb-60964e3d66c6" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913677    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913687    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913694    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89dd4ba-b869-4902-9cd3-968437773d1f" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913703    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b35c0e4-d1a3-4528-924b-1f095099ee53" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913711    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3de473-ca1b-4cdd-9696-d291633413b2" containerName="payload-pre-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913722    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913730    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8627883-0bb5-4416-a42a-f0c5c9ef4252" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913740    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="c445bad5-3c95-4631-b207-996f78b0d51e" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913749    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb747ca-ec06-45af-986d-dc4582cd3583" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913757    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac9525b-5d5a-4318-89c9-6cb675f8ad81" containerName="cleanup"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913767    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913774    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e843ff-8d4f-4276-9db2-83febf7e039b" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913781    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd31a8-8e83-4da8-9da8-bcac2d804f0c" containerName="cleanup"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913791    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4774c8e-c4c5-4229-8396-9682397df7b7" containerName="cleanup"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913798    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="266b0b26-9296-4a87-aa4e-a7ef5d3104ab" containerName="payload-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913806    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00600f9-826b-48ba-b576-dfe4e299957c" containerName="payload-pre-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913814    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="668a3983-44bd-4f91-954d-7c501263b172" containerName="maas-api"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913824    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8cd813-2c64-4ce0-876b-2aabb23af795" containerName="maas-api"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913833    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913842    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913852    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913860    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b96b271-e20b-4178-a7c5-8290d0aed2b5" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913868    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="428d296d-b09c-4d16-8486-7f12791481bb" containerName="maas-api"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913876    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1e6ce0-a826-4a6c-b6ac-75396e9825f9" containerName="payload-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913884    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce77fd3-d98f-4ac4-b8ba-562e561c8dff" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913893    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="131361f6-42d9-40a0-adb7-927c17ecf9b0" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913902    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa654b1a-05f5-4154-b259-bc74b75cc1c9" containerName="payload-pre-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913910    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="39620b2e-b2d5-4162-8115-d40e81939478" containerName="payload-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913919    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8434e7-5ef7-40f0-a981-cdbfb1e287b5" containerName="payload-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913929    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="517b403f-ed14-4aa2-8aee-3943ac901a12" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913939    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2361be4-77ae-456b-99f6-80f3c5aa0aed" containerName="maas-api"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913950    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="206918be-ffb7-4338-bd7f-f4950591052e" containerName="cleanup"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913960    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="415ade92-351b-4c3f-b432-3d43af90403d" containerName="payload-pre-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913969    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="84972876-2a60-4466-b92d-0a21a7237bd2" containerName="cleanup"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913978    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11e6b5c-300c-48e3-a427-446eddd74f52" containerName="main"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913987    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.913997    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a5e36-a7c0-44d5-b987-a79a18bac90b" containerName="payload-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.914006    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf262dc-2456-41aa-a3d1-5bcd7126583d" containerName="payload-pre-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.914015    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9710500-d6bf-45c1-8724-8dcf08b046a5" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.914024    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b5226a-d141-496c-8a53-1a29728f7590" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.914032    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65523bf-94e1-48f5-8fe1-37baaf65636c" containerName="payload-processing"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.914039    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc82122b-3adb-4448-897a-4165d1c711b1" containerName="istio-proxy"
Nov 03 04:26:39.916660 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.914048    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a63b964-d5a5-489d-becb-6c7fd95ec450" containerName="payload-pre-processing"
Nov 03 04:26:39.918761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.918744    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:39.972088 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:39.972060    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-246eeedd-openshift-default-dockercfg-txwt5\""
Nov 03 04:26:40.013832 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.013786    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:40.016990 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.016969    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017074 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017020    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017144    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4f8\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017227 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017201    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017297    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017452 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017334    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017582 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017562    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017674 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017601    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.017674 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.017619    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.024131 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.024105    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"]
Nov 03 04:26:40.118355 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118314    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118376    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4f8\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118419    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118451    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118485    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118543    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118882 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118612    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118882 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118641    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.118882 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118676    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.119056 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.118988    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.119056 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.119041    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.119167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.119094    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.119167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.119123    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.119276 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.119257    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.120902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.120881    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.121218 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.121200    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.180766 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.180711    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.180766 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.180746    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4f8\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8\") pod \"e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") " pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.231093 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.231072    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:40.294842 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.294749    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:40.666423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.666357    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"]
Nov 03 04:26:40.668741 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:26:40.668715    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1140a413_6543_4a5f_8731_26d35041a0c1.slice/crio-a902c7edaff9c0070a560c908c9f43856d15f7eb02948213a216d84b09af7e04 WatchSource:0}: Error finding container a902c7edaff9c0070a560c908c9f43856d15f7eb02948213a216d84b09af7e04: Status 404 returned error can't find the container with id a902c7edaff9c0070a560c908c9f43856d15f7eb02948213a216d84b09af7e04
Nov 03 04:26:40.670930 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.670894    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:26:40.671045 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.670976    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:26:40.671045 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.671020    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:26:40.841341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.841309    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" event={"ID":"1140a413-6543-4a5f-8731-26d35041a0c1","Type":"ContainerStarted","Data":"27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6"}
Nov 03 04:26:40.841524 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.841358    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" event={"ID":"1140a413-6543-4a5f-8731-26d35041a0c1","Type":"ContainerStarted","Data":"a902c7edaff9c0070a560c908c9f43856d15f7eb02948213a216d84b09af7e04"}
Nov 03 04:26:40.965221 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:40.965113    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podStartSLOduration=1.965096679 podStartE2EDuration="1.965096679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:26:40.964468649 +0000 UTC m=+5092.195361066" watchObservedRunningTime="2026-11-03 04:26:40.965096679 +0000 UTC m=+5092.195989097"
Nov 03 04:26:41.231703 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:41.231623    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:41.236801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:41.236778    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:41.846504 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:41.846472    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:41.847438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:41.847420    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:26:42.987135 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:42.987095    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:43.154849 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:43.154818    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:43.309042 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:43.308964    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:46.258756 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.258719    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:46.421251 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.421220    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:46.827997 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.827967    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"]
Nov 03 04:26:46.828365 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.828350    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy"
Nov 03 04:26:46.828443 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.828369    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8623e33-0051-4ac5-b370-53fc00aa6fe6" containerName="istio-proxy"
Nov 03 04:26:46.828443 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.828382    5515 cpu_manager.go:414] "containerMap: removing container" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy"
Nov 03 04:26:46.828443 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.828386    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8a6bb3-970e-4fce-bcad-01233bf889fc" containerName="istio-proxy"
Nov 03 04:26:46.828680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.828457    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c56464-7dbd-411a-a63f-14fec9ea0c58" containerName="istio-proxy"
Nov 03 04:26:46.831764 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.831746    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:46.893208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.893183    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"]
Nov 03 04:26:46.935487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.935460    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-246eeedd\""
Nov 03 04:26:46.935640 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.935499    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-246eeedd-dockercfg-zl7g7\""
Nov 03 04:26:46.980907 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.980877    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthlc\" (UniqueName: \"kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc\") pod \"payload-processing-e2e-ait-246eeedd-5578589-bvqmf\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") " pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:46.981030 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:46.980937    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume\") pod \"payload-processing-e2e-ait-246eeedd-5578589-bvqmf\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") " pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.081869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.081807    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthlc\" (UniqueName: \"kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc\") pod \"payload-processing-e2e-ait-246eeedd-5578589-bvqmf\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") " pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.081869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.081860    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume\") pod \"payload-processing-e2e-ait-246eeedd-5578589-bvqmf\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") " pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.082473 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.082455    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume\") pod \"payload-processing-e2e-ait-246eeedd-5578589-bvqmf\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") " pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.143503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.143474    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthlc\" (UniqueName: \"kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc\") pod \"payload-processing-e2e-ait-246eeedd-5578589-bvqmf\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") " pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.144253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.144235    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.515267 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.515242    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"]
Nov 03 04:26:47.516799 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:26:47.516771    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346643ad_1ef1_44b2_9ebd_256eee52c491.slice/crio-fa39848adb242d0e5b9fb9d9eeff6d854fb0b0da1d31f5f4fd82bc1d4726e406 WatchSource:0}: Error finding container fa39848adb242d0e5b9fb9d9eeff6d854fb0b0da1d31f5f4fd82bc1d4726e406: Status 404 returned error can't find the container with id fa39848adb242d0e5b9fb9d9eeff6d854fb0b0da1d31f5f4fd82bc1d4726e406
Nov 03 04:26:47.873061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.873025    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf" event={"ID":"346643ad-1ef1-44b2-9ebd-256eee52c491","Type":"ContainerStarted","Data":"76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8"}
Nov 03 04:26:47.873261 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.873075    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf" event={"ID":"346643ad-1ef1-44b2-9ebd-256eee52c491","Type":"ContainerStarted","Data":"fa39848adb242d0e5b9fb9d9eeff6d854fb0b0da1d31f5f4fd82bc1d4726e406"}
Nov 03 04:26:47.873261 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.873096    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:47.991494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:47.991437    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf" podStartSLOduration=1.991424616 podStartE2EDuration="1.991424616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:26:47.990814385 +0000 UTC m=+5099.221706803" watchObservedRunningTime="2026-11-03 04:26:47.991424616 +0000 UTC m=+5099.222317034"
Nov 03 04:26:49.459751 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.459722    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"]
Nov 03 04:26:49.463277 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.463259    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.516447 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.516423    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-246eeedd-serving-cert\""
Nov 03 04:26:49.568729 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.568702    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"]
Nov 03 04:26:49.603791 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.603750    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.603955 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.603869    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4ld\" (UniqueName: \"kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.604048 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.603988    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.604048 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.604017    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.658772 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.658742    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.715936    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.716042    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4ld\" (UniqueName: \"kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.716155    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.716182    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.717644    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:26:49.718017    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-serving-cert: secret "maas-api-e2e-ait-246eeedd-serving-cert" not found
Nov 03 04:26:49.719399 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:26:49.718134    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls podName:096e0404-6c46-4813-92e7-e1540e9036df nodeName:}" failed. No retries permitted until 2026-11-03 04:26:50.218114699 +0000 UTC m=+5101.449007108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls") pod "maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" (UID: "096e0404-6c46-4813-92e7-e1540e9036df") : secret "maas-api-e2e-ait-246eeedd-serving-cert" not found
Nov 03 04:26:49.720954 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.720923    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.780000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.779971    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4ld\" (UniqueName: \"kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:49.818784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.818761    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:49.961132 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:49.961102    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:50.221049 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.221015    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:50.223538 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.223519    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\") pod \"maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:50.375418 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.375390    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:50.417308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.417285    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:26:50.420953 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.420934    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:26:50.466459 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.466438    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:26:50.468893 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.468872    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:26:50.755744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.755722    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"]
Nov 03 04:26:50.758301 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:26:50.758275    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096e0404_6c46_4813_92e7_e1540e9036df.slice/crio-bd1a6dc4ea6bf2388b3e06e06798fe6059b0ca37c14119c407f9563a90d64464 WatchSource:0}: Error finding container bd1a6dc4ea6bf2388b3e06e06798fe6059b0ca37c14119c407f9563a90d64464: Status 404 returned error can't find the container with id bd1a6dc4ea6bf2388b3e06e06798fe6059b0ca37c14119c407f9563a90d64464
Nov 03 04:26:50.759728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.759707    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:26:50.892335 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:50.892301    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" event={"ID":"096e0404-6c46-4813-92e7-e1540e9036df","Type":"ContainerStarted","Data":"bd1a6dc4ea6bf2388b3e06e06798fe6059b0ca37c14119c407f9563a90d64464"}
Nov 03 04:26:50.984594 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:26:50.984533    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:26:51.894923 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:51.894888    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" event={"ID":"096e0404-6c46-4813-92e7-e1540e9036df","Type":"ContainerStarted","Data":"6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb"}
Nov 03 04:26:51.895539 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:51.894955    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:52.021783 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:52.021732    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" podStartSLOduration=2.131179367 podStartE2EDuration="3.021716637s" totalImagesPullingTime="890.53727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:26:52.020127257 +0000 UTC m=+5103.251019698" watchObservedRunningTime="2026-11-03 04:26:52.021716637 +0000 UTC m=+5103.252609120"
Nov 03 04:26:54.613213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:54.613179    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:54.775268 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:54.775219    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:26:57.888112 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:57.888079    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:26:57.905753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:57.905728    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:26:58.893315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:58.893285    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:26:59.738191 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:26:59.738154    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:04.639668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:04.639630    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:06.965449 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:06.965417    5515 generic.go:356] "Generic (PLEG): container finished" podID="1140a413-6543-4a5f-8731-26d35041a0c1" containerID="27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6" exitCode=0
Nov 03 04:27:06.966035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:06.965494    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" event={"ID":"1140a413-6543-4a5f-8731-26d35041a0c1","Type":"ContainerDied","Data":"27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6"}
Nov 03 04:27:06.966035 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:06.965937    5515 scope.go:122] "RemoveContainer" containerID="27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6"
Nov 03 04:27:06.967273 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:06.967243    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:27:06.967333 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:06.967317    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:27:06.967377 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:06.967351    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:27:07.992838 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:07.992796    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" event={"ID":"1140a413-6543-4a5f-8731-26d35041a0c1","Type":"ContainerStarted","Data":"856334ed09a4ccb1b7d1123951dba06756a4042ea4da16a21cabd3a8f6291fe2"}
Nov 03 04:27:08.232138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:08.232102    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:27:08.233165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:08.233144    5515 patch_prober.go:29] interesting pod/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 04:27:08.233225 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:08.233190    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 04:27:09.231737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:09.231702    5515 patch_prober.go:29] interesting pod/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 04:27:09.232144 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:09.231754    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 04:27:10.076136 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:10.076103    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:10.231601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:10.231529    5515 patch_prober.go:29] interesting pod/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 04:27:10.231783 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:10.231632    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 04:27:11.231523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:11.231489    5515 patch_prober.go:29] interesting pod/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 04:27:11.231877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:11.231584    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 04:27:11.846753 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:11.846716    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:27:12.231761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:12.231727    5515 patch_prober.go:29] interesting pod/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 04:27:12.231976 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:12.231782    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 04:27:13.236227 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:13.236198    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:27:13.237247 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:13.237230    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:27:15.039324 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:15.039287    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:17.487591 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:17.487515    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:20.049831 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:20.049788    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:20.983851 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:27:20.983812    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:27:24.948707 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:24.948620    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:29.962207 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:29.962171    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:31.092225 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:31.092181    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"]
Nov 03 04:27:31.455154 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:31.455117    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:31.599194 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:31.599149    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:34.859112 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:34.859064    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:36.093322 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.093279    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf" podUID="346643ad-1ef1-44b2-9ebd-256eee52c491" containerName="payload-processing" containerID="cri-o://76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8" gracePeriod=25
Nov 03 04:27:36.342724 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.342699    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:27:36.467965 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.467934    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc\" (UniqueName: \"kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc\") pod \"346643ad-1ef1-44b2-9ebd-256eee52c491\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") "
Nov 03 04:27:36.468187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.467991    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume\") pod \"346643ad-1ef1-44b2-9ebd-256eee52c491\" (UID: \"346643ad-1ef1-44b2-9ebd-256eee52c491\") "
Nov 03 04:27:36.468398 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.468376    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume" pod "346643ad-1ef1-44b2-9ebd-256eee52c491" (UID: "346643ad-1ef1-44b2-9ebd-256eee52c491"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:27:36.470252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.470230    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc" pod "346643ad-1ef1-44b2-9ebd-256eee52c491" (UID: "346643ad-1ef1-44b2-9ebd-256eee52c491"). InnerVolumeSpecName "kube-api-access-lthlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:27:36.569470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.569443    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lthlc\" (UniqueName: \"kubernetes.io/projected/346643ad-1ef1-44b2-9ebd-256eee52c491-kube-api-access-lthlc\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:36.569470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:36.569470    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/346643ad-1ef1-44b2-9ebd-256eee52c491-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:37.123482 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.123444    5515 generic.go:356] "Generic (PLEG): container finished" podID="346643ad-1ef1-44b2-9ebd-256eee52c491" containerID="76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8" exitCode=0
Nov 03 04:27:37.124009 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.123502    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"
Nov 03 04:27:37.124009 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.123534    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf" event={"ID":"346643ad-1ef1-44b2-9ebd-256eee52c491","Type":"ContainerDied","Data":"76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8"}
Nov 03 04:27:37.124009 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.123655    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf" event={"ID":"346643ad-1ef1-44b2-9ebd-256eee52c491","Type":"ContainerDied","Data":"fa39848adb242d0e5b9fb9d9eeff6d854fb0b0da1d31f5f4fd82bc1d4726e406"}
Nov 03 04:27:37.124009 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.123691    5515 scope.go:122] "RemoveContainer" containerID="76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8"
Nov 03 04:27:37.132257 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.132242    5515 scope.go:122] "RemoveContainer" containerID="76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8"
Nov 03 04:27:37.132500 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:27:37.132480    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8\": container with ID starting with 76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8 not found: ID does not exist" containerID="76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8"
Nov 03 04:27:37.132561 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.132507    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8"} err="failed to get container status \"76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8\": rpc error: code = NotFound desc = could not find container \"76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8\": container with ID starting with 76be277eb88f208de560e89ba7aa1e0e944332f59973a91026620403c1af4be8 not found: ID does not exist"
Nov 03 04:27:37.294001 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.293924    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"]
Nov 03 04:27:37.343436 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:37.343404    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-246eeedd-5578589-bvqmf"]
Nov 03 04:27:38.892781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:38.892740    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346643ad-1ef1-44b2-9ebd-256eee52c491" path="/var/lib/kubelet/pods/346643ad-1ef1-44b2-9ebd-256eee52c491/volumes"
Nov 03 04:27:39.119961 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.119926    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"]
Nov 03 04:27:39.120162 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.120142    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" podUID="096e0404-6c46-4813-92e7-e1540e9036df" containerName="maas-api" containerID="cri-o://6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb" gracePeriod=30
Nov 03 04:27:39.244490 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.244464    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:39.375759 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.375730    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:27:39.500860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.500780    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\") pod \"096e0404-6c46-4813-92e7-e1540e9036df\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") "
Nov 03 04:27:39.500860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.500830    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld\" (UniqueName: \"kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld\") pod \"096e0404-6c46-4813-92e7-e1540e9036df\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") "
Nov 03 04:27:39.501110 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.501005    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs\") pod \"096e0404-6c46-4813-92e7-e1540e9036df\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") "
Nov 03 04:27:39.501110 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.501053    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca\") pod \"096e0404-6c46-4813-92e7-e1540e9036df\" (UID: \"096e0404-6c46-4813-92e7-e1540e9036df\") "
Nov 03 04:27:39.501856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.501828    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca" pod "096e0404-6c46-4813-92e7-e1540e9036df" (UID: "096e0404-6c46-4813-92e7-e1540e9036df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:27:39.503459 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.503433    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls" pod "096e0404-6c46-4813-92e7-e1540e9036df" (UID: "096e0404-6c46-4813-92e7-e1540e9036df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:27:39.503629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.503591    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld" pod "096e0404-6c46-4813-92e7-e1540e9036df" (UID: "096e0404-6c46-4813-92e7-e1540e9036df"). InnerVolumeSpecName "kube-api-access-tp4ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:27:39.504236 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.504204    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs" pod "096e0404-6c46-4813-92e7-e1540e9036df" (UID: "096e0404-6c46-4813-92e7-e1540e9036df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:27:39.601787 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.601751    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:39.601787 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.601781    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096e0404-6c46-4813-92e7-e1540e9036df-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:39.601787 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.601791    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096e0404-6c46-4813-92e7-e1540e9036df-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:39.602052 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:39.601800    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp4ld\" (UniqueName: \"kubernetes.io/projected/096e0404-6c46-4813-92e7-e1540e9036df-kube-api-access-tp4ld\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:40.140741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.140708    5515 generic.go:356] "Generic (PLEG): container finished" podID="096e0404-6c46-4813-92e7-e1540e9036df" containerID="6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb" exitCode=0
Nov 03 04:27:40.141208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.140762    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"
Nov 03 04:27:40.141208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.140773    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" event={"ID":"096e0404-6c46-4813-92e7-e1540e9036df","Type":"ContainerDied","Data":"6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb"}
Nov 03 04:27:40.141208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.140815    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f" event={"ID":"096e0404-6c46-4813-92e7-e1540e9036df","Type":"ContainerDied","Data":"bd1a6dc4ea6bf2388b3e06e06798fe6059b0ca37c14119c407f9563a90d64464"}
Nov 03 04:27:40.141208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.140837    5515 scope.go:122] "RemoveContainer" containerID="6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb"
Nov 03 04:27:40.150292 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.150273    5515 scope.go:122] "RemoveContainer" containerID="6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb"
Nov 03 04:27:40.150601 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:27:40.150578    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb\": container with ID starting with 6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb not found: ID does not exist" containerID="6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb"
Nov 03 04:27:40.150658 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.150610    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb"} err="failed to get container status \"6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb\": rpc error: code = NotFound desc = could not find container \"6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb\": container with ID starting with 6b937c7e264ef8cf1f73f313a1f475ef6710dfbd285de67e8ec8dbbb30fe67bb not found: ID does not exist"
Nov 03 04:27:40.308559 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.308521    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"]
Nov 03 04:27:40.313378 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.313354    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-246eeedd-5c5c54dfc5-8fh4f"]
Nov 03 04:27:40.800519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.800483    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:40.893253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:40.893211    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096e0404-6c46-4813-92e7-e1540e9036df" path="/var/lib/kubelet/pods/096e0404-6c46-4813-92e7-e1540e9036df/volumes"
Nov 03 04:27:41.109359 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:41.109316    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:43.809429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:43.809394    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:27:45.968526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:45.968488    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"]
Nov 03 04:27:45.968947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:45.968705    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy" containerID="cri-o://856334ed09a4ccb1b7d1123951dba06756a4042ea4da16a21cabd3a8f6291fe2" gracePeriod=30
Nov 03 04:27:45.981499 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:45.981473    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:27:50.985200 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:27:50.985147    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:27:51.202210 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.202181    5515 generic.go:356] "Generic (PLEG): container finished" podID="1140a413-6543-4a5f-8731-26d35041a0c1" containerID="856334ed09a4ccb1b7d1123951dba06756a4042ea4da16a21cabd3a8f6291fe2" exitCode=0
Nov 03 04:27:51.202354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.202247    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" event={"ID":"1140a413-6543-4a5f-8731-26d35041a0c1","Type":"ContainerDied","Data":"856334ed09a4ccb1b7d1123951dba06756a4042ea4da16a21cabd3a8f6291fe2"}
Nov 03 04:27:51.202354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.202317    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47" event={"ID":"1140a413-6543-4a5f-8731-26d35041a0c1","Type":"ContainerDied","Data":"a902c7edaff9c0070a560c908c9f43856d15f7eb02948213a216d84b09af7e04"}
Nov 03 04:27:51.202354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.202341    5515 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a902c7edaff9c0070a560c908c9f43856d15f7eb02948213a216d84b09af7e04"
Nov 03 04:27:51.202619 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.202360    5515 scope.go:122] "RemoveContainer" containerID="27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6"
Nov 03 04:27:51.214460 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:27:51.214438    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6\": container with ID starting with 27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6 not found: ID does not exist" containerID="27b72c53306389ede2a542da57a8faeda508973a1120c7e1cad3dff1d7d7acd6"
Nov 03 04:27:51.217859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.217841    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:27:51.299978 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.299898    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.299978 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.299938    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.299978 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.299956    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.299991    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300032    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300058    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300101    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300155    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300192    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket\") pod \"1140a413-6543-4a5f-8731-26d35041a0c1\" (UID: \"1140a413-6543-4a5f-8731-26d35041a0c1\") "
Nov 03 04:27:51.300264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300254    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:27:51.300711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300280    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:27:51.300711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300580    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.300711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300604    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.300711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300580    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:27:51.300711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300643    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:27:51.301151 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.300729    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:27:51.302497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.302468    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "kube-api-access-qx4f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:27:51.302612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.302536    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:27:51.302874 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.302849    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:27:51.302874 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.302865    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token" pod "1140a413-6543-4a5f-8731-26d35041a0c1" (UID: "1140a413-6543-4a5f-8731-26d35041a0c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:27:51.401857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401826    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.401857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401854    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.401857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401863    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1140a413-6543-4a5f-8731-26d35041a0c1-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.402098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401873    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4f8\" (UniqueName: \"kubernetes.io/projected/1140a413-6543-4a5f-8731-26d35041a0c1-kube-api-access-qx4f8\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.402098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401890    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1140a413-6543-4a5f-8731-26d35041a0c1-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.402098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401898    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:51.402098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:51.401906    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1140a413-6543-4a5f-8731-26d35041a0c1-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:27:52.207913 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.207882    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"
Nov 03 04:27:52.449791 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.449747    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"]
Nov 03 04:27:52.500480 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.500399    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-246eeedd-openshift-default-68765d5d7f-9kk47"]
Nov 03 04:27:52.529815 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.529778    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7957588c98-vqmlg"]
Nov 03 04:27:52.530320 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530302    5515 cpu_manager.go:414] "containerMap: removing container" podUID="096e0404-6c46-4813-92e7-e1540e9036df" containerName="maas-api"
Nov 03 04:27:52.530386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530321    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096e0404-6c46-4813-92e7-e1540e9036df" containerName="maas-api"
Nov 03 04:27:52.530386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530337    5515 cpu_manager.go:414] "containerMap: removing container" podUID="346643ad-1ef1-44b2-9ebd-256eee52c491" containerName="payload-processing"
Nov 03 04:27:52.530386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530345    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346643ad-1ef1-44b2-9ebd-256eee52c491" containerName="payload-processing"
Nov 03 04:27:52.530386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530380    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy"
Nov 03 04:27:52.530386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530387    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy"
Nov 03 04:27:52.530578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530397    5515 cpu_manager.go:414] "containerMap: removing container" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy"
Nov 03 04:27:52.530578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530402    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy"
Nov 03 04:27:52.530578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530486    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy"
Nov 03 04:27:52.530578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530495    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" containerName="istio-proxy"
Nov 03 04:27:52.530578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530503    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="096e0404-6c46-4813-92e7-e1540e9036df" containerName="maas-api"
Nov 03 04:27:52.530578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.530510    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="346643ad-1ef1-44b2-9ebd-256eee52c491" containerName="payload-processing"
Nov 03 04:27:52.534935 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.534920    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.594081 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.594036    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 04:27:52.594282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.594138    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 04:27:52.594282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.594197    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 04:27:52.594282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.594250    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4f44h\""
Nov 03 04:27:52.612116 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.612083    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.612288 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.612204    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.612358 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.612291    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqx74\" (UniqueName: \"kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.614578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.614532    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7957588c98-vqmlg"]
Nov 03 04:27:52.713766 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.713731    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.713960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.713794    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx74\" (UniqueName: \"kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.713960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.713865    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.716533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.716512    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.716609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.716577    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.783319 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.783245    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqx74\" (UniqueName: \"kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74\") pod \"maas-controller-7957588c98-vqmlg\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") " pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.847533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.847506    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:52.892475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:52.892443    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1140a413-6543-4a5f-8731-26d35041a0c1" path="/var/lib/kubelet/pods/1140a413-6543-4a5f-8731-26d35041a0c1/volumes"
Nov 03 04:27:53.225869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:53.225769    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7957588c98-vqmlg"]
Nov 03 04:27:53.228192 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:27:53.228165    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f94c6d9_7f68_404c_9763_a14d53684cbf.slice/crio-5e2da9fed22cc6f6ca3e4f2d6a4a87b79bf98aa96ad00d39ce0629e2f4f1f9cc WatchSource:0}: Error finding container 5e2da9fed22cc6f6ca3e4f2d6a4a87b79bf98aa96ad00d39ce0629e2f4f1f9cc: Status 404 returned error can't find the container with id 5e2da9fed22cc6f6ca3e4f2d6a4a87b79bf98aa96ad00d39ce0629e2f4f1f9cc
Nov 03 04:27:54.223994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:54.223956    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-vqmlg" event={"ID":"0f94c6d9-7f68-404c-9763-a14d53684cbf","Type":"ContainerStarted","Data":"5e2da9fed22cc6f6ca3e4f2d6a4a87b79bf98aa96ad00d39ce0629e2f4f1f9cc"}
Nov 03 04:27:56.235392 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:56.235352    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-vqmlg" event={"ID":"0f94c6d9-7f68-404c-9763-a14d53684cbf","Type":"ContainerStarted","Data":"19922bbb6340c9425ff78c8032cc9ff5eae8ee5c6519dab7a02430eafc7b494b"}
Nov 03 04:27:56.235836 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:56.235510    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:27:56.371581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:27:56.371505    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7957588c98-vqmlg" podStartSLOduration=2.190918087 podStartE2EDuration="4.371491774s" totalImagesPullingTime="2.180573687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:27:56.368822266 +0000 UTC m=+5167.599714684" watchObservedRunningTime="2026-11-03 04:27:56.371491774 +0000 UTC m=+5167.602384212"
Nov 03 04:28:00.437008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:00.436975    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:00.629494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:00.629464    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:07.242664 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:07.242622    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:28:08.058836 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.058803    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"]
Nov 03 04:28:08.062492 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.062471    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.109542 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.109509    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"]
Nov 03 04:28:08.113023 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.113005    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"]
Nov 03 04:28:08.113146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.113127    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.116165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.116147    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.149732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149708    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34655d27-700c-4347-beac-dfb690ec4e7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.149860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149741    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54478bf9-f992-496c-b32c-979cea64a63c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.149860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149763    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34655d27-700c-4347-beac-dfb690ec4e7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.149860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149780    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149927    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149960    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.149992    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150040    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150146 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150114    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54478bf9-f992-496c-b32c-979cea64a63c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150155    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64240ef-e871-4000-ad47-b8343dc61b53-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150214    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjs5f\" (UniqueName: \"kubernetes.io/projected/34655d27-700c-4347-beac-dfb690ec4e7b-kube-api-access-mjs5f\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150232    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150246    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150265    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150283    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150330    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lc9g\" (UniqueName: \"kubernetes.io/projected/f64240ef-e871-4000-ad47-b8343dc61b53-kube-api-access-6lc9g\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150371 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150363    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2x4p\" (UniqueName: \"kubernetes.io/projected/54478bf9-f992-496c-b32c-979cea64a63c-kube-api-access-k2x4p\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150391    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54478bf9-f992-496c-b32c-979cea64a63c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150445    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150465    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64240ef-e871-4000-ad47-b8343dc61b53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150482    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150501    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150569    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.150744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150606    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150624    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.150744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150670    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34655d27-700c-4347-beac-dfb690ec4e7b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.150744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.150698    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64240ef-e871-4000-ad47-b8343dc61b53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.159112 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.159093    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gfd7r\""
Nov 03 04:28:08.160902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.160882    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"]
Nov 03 04:28:08.213115 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.213081    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"]
Nov 03 04:28:08.213282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.213155    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"]
Nov 03 04:28:08.251249 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251223    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251254    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251274    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34655d27-700c-4347-beac-dfb690ec4e7b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251288    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64240ef-e871-4000-ad47-b8343dc61b53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251317    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34655d27-700c-4347-beac-dfb690ec4e7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251350    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54478bf9-f992-496c-b32c-979cea64a63c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251487    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34655d27-700c-4347-beac-dfb690ec4e7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251528    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251606    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251635    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251663    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251691    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.251710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251716    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54478bf9-f992-496c-b32c-979cea64a63c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251741    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64240ef-e871-4000-ad47-b8343dc61b53-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251785    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjs5f\" (UniqueName: \"kubernetes.io/projected/34655d27-700c-4347-beac-dfb690ec4e7b-kube-api-access-mjs5f\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251813    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251839    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251867    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251905    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251937    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lc9g\" (UniqueName: \"kubernetes.io/projected/f64240ef-e871-4000-ad47-b8343dc61b53-kube-api-access-6lc9g\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251969    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2x4p\" (UniqueName: \"kubernetes.io/projected/54478bf9-f992-496c-b32c-979cea64a63c-kube-api-access-k2x4p\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.251997    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54478bf9-f992-496c-b32c-979cea64a63c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252009    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252016    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252091    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252129    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64240ef-e871-4000-ad47-b8343dc61b53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252149    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34655d27-700c-4347-beac-dfb690ec4e7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252157    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252194    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252232    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.252672 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252638    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.253687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252783    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.253687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.252870    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.253687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253028    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.253687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253404    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.253687 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253452    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54478bf9-f992-496c-b32c-979cea64a63c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.253947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253728    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.253947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253733    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.253947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253781    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.254102 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.253980    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.254340 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.254316    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.254450 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.254366    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64240ef-e871-4000-ad47-b8343dc61b53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.254662 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.254642    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34655d27-700c-4347-beac-dfb690ec4e7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.254750 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.254721    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34655d27-700c-4347-beac-dfb690ec4e7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.254999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.254978    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64240ef-e871-4000-ad47-b8343dc61b53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.255190 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.255169    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54478bf9-f992-496c-b32c-979cea64a63c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.256047 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.256018    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54478bf9-f992-496c-b32c-979cea64a63c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.256435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.256417    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64240ef-e871-4000-ad47-b8343dc61b53-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.313934 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.313870    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34655d27-700c-4347-beac-dfb690ec4e7b-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.314114 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.314094    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54478bf9-f992-496c-b32c-979cea64a63c-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.317160 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.317136    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64240ef-e871-4000-ad47-b8343dc61b53-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.317620 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.317600    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjs5f\" (UniqueName: \"kubernetes.io/projected/34655d27-700c-4347-beac-dfb690ec4e7b-kube-api-access-mjs5f\") pod \"maas-default-gateway-openshift-default-76f4876667-nv7sc\" (UID: \"34655d27-700c-4347-beac-dfb690ec4e7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.318639 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.318622    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lc9g\" (UniqueName: \"kubernetes.io/projected/f64240ef-e871-4000-ad47-b8343dc61b53-kube-api-access-6lc9g\") pod \"maas-default-gateway-openshift-default-76f4876667-58f56\" (UID: \"f64240ef-e871-4000-ad47-b8343dc61b53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.321344 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.321326    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2x4p\" (UniqueName: \"kubernetes.io/projected/54478bf9-f992-496c-b32c-979cea64a63c-kube-api-access-k2x4p\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbt8\" (UID: \"54478bf9-f992-496c-b32c-979cea64a63c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.375083 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.375058    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:08.426494 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.426288    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:08.432817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.432788    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:08.800460 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.800436    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"]
Nov 03 04:28:08.802038 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:28:08.802012    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34655d27_700c_4347_beac_dfb690ec4e7b.slice/crio-a184197dd1af7b1451ed103f79811cdddc03e2c0c61d96db45a1245118785a00 WatchSource:0}: Error finding container a184197dd1af7b1451ed103f79811cdddc03e2c0c61d96db45a1245118785a00: Status 404 returned error can't find the container with id a184197dd1af7b1451ed103f79811cdddc03e2c0c61d96db45a1245118785a00
Nov 03 04:28:08.804194 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.804162    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.804293 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.804236    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.804293 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.804265    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.855843 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.855571    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"]
Nov 03 04:28:08.864165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.864130    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.864284 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.864204    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.864284 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.864247    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.902158 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.902109    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"]
Nov 03 04:28:08.905926 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:28:08.905892    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64240ef_e871_4000_ad47_b8343dc61b53.slice/crio-b07850c1ca8080484d9b04775bd2f291be0fe26b608ed01333cf49afb68be8dd WatchSource:0}: Error finding container b07850c1ca8080484d9b04775bd2f291be0fe26b608ed01333cf49afb68be8dd: Status 404 returned error can't find the container with id b07850c1ca8080484d9b04775bd2f291be0fe26b608ed01333cf49afb68be8dd
Nov 03 04:28:08.913128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.913092    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.913248 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.913160    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:08.913248 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:08.913192    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:28:09.311075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.311038    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" event={"ID":"54478bf9-f992-496c-b32c-979cea64a63c","Type":"ContainerStarted","Data":"74af91c3221ad5a530a9945b6e54e939458f7cef5d5bb08a68aadd8e0be8f042"}
Nov 03 04:28:09.319972 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.319926    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" event={"ID":"54478bf9-f992-496c-b32c-979cea64a63c","Type":"ContainerStarted","Data":"90c3a39ad6980fa77b2a01a2c8ced8393bf2a763354501d664cd6d736774d7ff"}
Nov 03 04:28:09.320849 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.320815    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" event={"ID":"34655d27-700c-4347-beac-dfb690ec4e7b","Type":"ContainerStarted","Data":"e187b10dc25bfc43725660b015932258daa6d0d3bcfebc1f0f6e62ec9d0616c9"}
Nov 03 04:28:09.321082 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.321061    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" event={"ID":"34655d27-700c-4347-beac-dfb690ec4e7b","Type":"ContainerStarted","Data":"a184197dd1af7b1451ed103f79811cdddc03e2c0c61d96db45a1245118785a00"}
Nov 03 04:28:09.325264 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.325232    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" event={"ID":"f64240ef-e871-4000-ad47-b8343dc61b53","Type":"ContainerStarted","Data":"48f0423aa3f7e9f639edb614bc63db4dd61aef4f8fe60f1eea320cfe07635213"}
Nov 03 04:28:09.325423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.325274    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" event={"ID":"f64240ef-e871-4000-ad47-b8343dc61b53","Type":"ContainerStarted","Data":"b07850c1ca8080484d9b04775bd2f291be0fe26b608ed01333cf49afb68be8dd"}
Nov 03 04:28:09.393575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.389820    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:09.393575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.389973    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:09.393575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.390023    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:09.429568 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.426872    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:09.429568 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.427936    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:09.429568 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.427982    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:09.437565 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.436911    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:09.437565 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.437020    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:09.437565 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.437057    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:09.438574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.437982    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podStartSLOduration=1.437966315 podStartE2EDuration="1.437966315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:28:09.437361212 +0000 UTC m=+5180.668253634" watchObservedRunningTime="2026-11-03 04:28:09.437966315 +0000 UTC m=+5180.668858731"
Nov 03 04:28:09.723574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:09.711667    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podStartSLOduration=1.7116443220000002 podStartE2EDuration="1.711644322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:28:09.582045612 +0000 UTC m=+5180.812938031" watchObservedRunningTime="2026-11-03 04:28:09.711644322 +0000 UTC m=+5180.942536740"
Nov 03 04:28:10.376003 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:10.375964    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:10.376652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:10.376619    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:10.429575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:10.427061    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:10.429575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:10.427138    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:10.434944 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:10.434910    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:10.435178 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:10.435150    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:11.381574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:11.380963    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:11.381574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:11.381026    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:11.430097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:11.430011    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:11.430097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:11.430069    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:11.436710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:11.436674    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:11.436922 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:11.436898    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:12.382017 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:12.381977    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:12.382683 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:12.382648    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:12.428000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:12.427962    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:12.428312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:12.428280    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:12.436239 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:12.436060    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:12.436239 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:12.436128    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:13.383575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:13.383016    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:13.383575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:13.383081    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:13.427126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:13.426997    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:13.427126 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:13.427064    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:13.436383 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:13.436349    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:13.436702 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:13.436670    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:14.380040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:14.379988    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:14.380352 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:14.380322    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:14.427071 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:14.427032    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:14.427722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:14.427688    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:14.436003 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:14.435972    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:14.436259 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:14.436230    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:15.375642 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:15.375593    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:15.375962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:15.375665    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:15.427282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:15.427245    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:15.427925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:15.427303    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:15.433379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:15.433343    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:15.433517 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:15.433383    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:16.376285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:16.376196    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 03 04:28:16.376518 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:16.376319    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 03 04:28:16.427667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:16.427629    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 03 04:28:16.428203 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:16.427692    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 03 04:28:16.433386 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:16.433350    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body=
Nov 03 04:28:16.433528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:16.433409    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused"
Nov 03 04:28:17.738571 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.737861    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:17.739924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.739140    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:17.741598 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.741537    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8"
Nov 03 04:28:17.805458 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.805422    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:17.805894 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.805872    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:17.807097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.807076    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc"
Nov 03 04:28:17.848751 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.848714    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:17.849014 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.848996    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:17.849845 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.849827    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56"
Nov 03 04:28:17.884277 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.884237    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:17.934926 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:17.934870    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podStartSLOduration=9.934858193 podStartE2EDuration="9.934858193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:28:09.725667958 +0000 UTC m=+5180.956560359" watchObservedRunningTime="2026-11-03 04:28:17.934858193 +0000 UTC m=+5189.165750611"
Nov 03 04:28:21.047665 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:28:21.047623    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:28:27.109928 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.109891    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:27.255230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.255199    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7957588c98-vqmlg"]
Nov 03 04:28:27.255447 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.255390    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7957588c98-vqmlg" podUID="0f94c6d9-7f68-404c-9763-a14d53684cbf" containerName="manager" containerID="cri-o://19922bbb6340c9425ff78c8032cc9ff5eae8ee5c6519dab7a02430eafc7b494b" gracePeriod=30
Nov 03 04:28:27.446160 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.446122    5515 generic.go:356] "Generic (PLEG): container finished" podID="0f94c6d9-7f68-404c-9763-a14d53684cbf" containerID="19922bbb6340c9425ff78c8032cc9ff5eae8ee5c6519dab7a02430eafc7b494b" exitCode=0
Nov 03 04:28:27.446349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.446199    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-vqmlg" event={"ID":"0f94c6d9-7f68-404c-9763-a14d53684cbf","Type":"ContainerDied","Data":"19922bbb6340c9425ff78c8032cc9ff5eae8ee5c6519dab7a02430eafc7b494b"}
Nov 03 04:28:27.502780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.502756    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:28:27.592315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.592280    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert\") pod \"0f94c6d9-7f68-404c-9763-a14d53684cbf\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") "
Nov 03 04:28:27.592315 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.592317    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74\" (UniqueName: \"kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74\") pod \"0f94c6d9-7f68-404c-9763-a14d53684cbf\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") "
Nov 03 04:28:27.592637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.592342    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs\") pod \"0f94c6d9-7f68-404c-9763-a14d53684cbf\" (UID: \"0f94c6d9-7f68-404c-9763-a14d53684cbf\") "
Nov 03 04:28:27.594839 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.594808    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert" pod "0f94c6d9-7f68-404c-9763-a14d53684cbf" (UID: "0f94c6d9-7f68-404c-9763-a14d53684cbf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:28:27.595142 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.595108    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs" pod "0f94c6d9-7f68-404c-9763-a14d53684cbf" (UID: "0f94c6d9-7f68-404c-9763-a14d53684cbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:28:27.595267 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.595154    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74" pod "0f94c6d9-7f68-404c-9763-a14d53684cbf" (UID: "0f94c6d9-7f68-404c-9763-a14d53684cbf"). InnerVolumeSpecName "kube-api-access-dqx74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:28:27.693224 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.693140    5515 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-webhook-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:28:27.693224 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.693172    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqx74\" (UniqueName: \"kubernetes.io/projected/0f94c6d9-7f68-404c-9763-a14d53684cbf-kube-api-access-dqx74\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:28:27.693224 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:27.693184    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f94c6d9-7f68-404c-9763-a14d53684cbf-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:28:28.452406 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:28.452375    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-vqmlg"
Nov 03 04:28:28.452406 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:28.452385    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-vqmlg" event={"ID":"0f94c6d9-7f68-404c-9763-a14d53684cbf","Type":"ContainerDied","Data":"5e2da9fed22cc6f6ca3e4f2d6a4a87b79bf98aa96ad00d39ce0629e2f4f1f9cc"}
Nov 03 04:28:28.452945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:28.452440    5515 scope.go:122] "RemoveContainer" containerID="19922bbb6340c9425ff78c8032cc9ff5eae8ee5c6519dab7a02430eafc7b494b"
Nov 03 04:28:28.620756 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:28.620716    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7957588c98-vqmlg"]
Nov 03 04:28:28.671504 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:28.671467    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7957588c98-vqmlg"]
Nov 03 04:28:28.893016 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:28.892975    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f94c6d9-7f68-404c-9763-a14d53684cbf" path="/var/lib/kubelet/pods/0f94c6d9-7f68-404c-9763-a14d53684cbf/volumes"
Nov 03 04:28:29.951020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:29.950973    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:30.115198 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.115158    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7957588c98-phcw7"]
Nov 03 04:28:30.117128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.116845    5515 cpu_manager.go:414] "containerMap: removing container" podUID="0f94c6d9-7f68-404c-9763-a14d53684cbf" containerName="manager"
Nov 03 04:28:30.117128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.116877    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f94c6d9-7f68-404c-9763-a14d53684cbf" containerName="manager"
Nov 03 04:28:30.119195 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.117370    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f94c6d9-7f68-404c-9763-a14d53684cbf" containerName="manager"
Nov 03 04:28:30.121947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.121921    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.218654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.218543    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-metrics-certs\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.218654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.218632    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-webhook-cert\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.218856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.218792    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-kube-api-access-4wq95\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.227467 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.227434    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 04:28:30.227651 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.227470    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 04:28:30.227651 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.227469    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 04:28:30.227651 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.227491    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4f44h\""
Nov 03 04:28:30.229101 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.229082    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7957588c98-phcw7"]
Nov 03 04:28:30.320112 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.320070    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-kube-api-access-4wq95\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.320343 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.320170    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-metrics-certs\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.320343 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.320219    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-webhook-cert\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.322904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.322875    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-metrics-certs\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.323027 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.323007    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-webhook-cert\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.383076 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.383039    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wq95\" (UniqueName: \"kubernetes.io/projected/5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e-kube-api-access-4wq95\") pod \"maas-controller-7957588c98-phcw7\" (UID: \"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e\") " pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.435786 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.435740    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:30.812231 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:30.812202    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7957588c98-phcw7"]
Nov 03 04:28:30.813524 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:28:30.813503    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7483a5_8b8c_4890_8a6b_9f4a3aa8d08e.slice/crio-8cdee55126dceaf27d3966b5277817af37072f3c16ea533bfdc7dc8249a8907a WatchSource:0}: Error finding container 8cdee55126dceaf27d3966b5277817af37072f3c16ea533bfdc7dc8249a8907a: Status 404 returned error can't find the container with id 8cdee55126dceaf27d3966b5277817af37072f3c16ea533bfdc7dc8249a8907a
Nov 03 04:28:31.469397 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:31.469362    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-phcw7" event={"ID":"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e","Type":"ContainerStarted","Data":"8cdee55126dceaf27d3966b5277817af37072f3c16ea533bfdc7dc8249a8907a"}
Nov 03 04:28:32.474877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:32.474838    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7957588c98-phcw7" event={"ID":"5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e","Type":"ContainerStarted","Data":"d1c5b88bee68a208bd4632351de91b00b8cf1af90a4da71f73962f5d834955ee"}
Nov 03 04:28:32.475350 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:32.474899    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:32.597349 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:32.597298    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7957588c98-phcw7" podStartSLOduration=1.816771924 podStartE2EDuration="2.597284953s" totalImagesPullingTime="780.513029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:28:32.596332723 +0000 UTC m=+5203.827225142" watchObservedRunningTime="2026-11-03 04:28:32.597284953 +0000 UTC m=+5203.828177406"
Nov 03 04:28:43.374227 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:43.374193    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:43.481813 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:43.481780    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7957588c98-phcw7"
Nov 03 04:28:43.522590 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:43.522539    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:28:50.984616 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:28:50.984577    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:28:54.999075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:28:54.998986    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:29:09.759809 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:09.759759    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:29:10.076482 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:10.076395    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:29:14.108655 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:14.108620    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:29:18.182098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:18.182055    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:29:21.052033 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:29:21.051985    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:29:24.678175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:24.678139    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:29:34.692224 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:34.692190    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:29:43.517718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:29:43.517686    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:29:50.983440 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:29:50.983396    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:30:18.539205 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:18.539168    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:30:20.984188 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:30:20.984151    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:30:22.107653 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:22.107619    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:30:23.050962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:23.050921    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:30:23.210707 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:23.210671    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:30:25.388497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.388463    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"]
Nov 03 04:30:25.392099 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.392084    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.478134 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.478094    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:30:25.480912 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.480893    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-edfabb23-kserve-self-signed-certs\""
Nov 03 04:30:25.497710 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.497685    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"]
Nov 03 04:30:25.550783 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.550753    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.550991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.550806    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.550991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.550882    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.551079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.551022    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.551079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.551065    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.551170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.551150    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8hd\" (UniqueName: \"kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652260    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652313    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652351    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652411    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652451    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652515    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8hd\" (UniqueName: \"kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652784    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652837    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.652904 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.652863    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.654855 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.654832    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.655022 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.655004    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:25.714781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:25.714748    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8hd\" (UniqueName: \"kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd\") pod \"e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") " pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:26.003454 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:26.003374    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:26.213121 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:26.213061    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:30:26.381617 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:26.381593    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"]
Nov 03 04:30:26.383297 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:30:26.383269    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod651fd98c_04d6_4d7d_8cbd_a7551ea06e1e.slice/crio-32acae74176ade69718457b1897331265d96425ad3bea48921028f1330e7d8fc WatchSource:0}: Error finding container 32acae74176ade69718457b1897331265d96425ad3bea48921028f1330e7d8fc: Status 404 returned error can't find the container with id 32acae74176ade69718457b1897331265d96425ad3bea48921028f1330e7d8fc
Nov 03 04:30:27.139995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:27.139954    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" event={"ID":"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e","Type":"ContainerStarted","Data":"a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44"}
Nov 03 04:30:27.139995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:27.140010    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" event={"ID":"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e","Type":"ContainerStarted","Data":"32acae74176ade69718457b1897331265d96425ad3bea48921028f1330e7d8fc"}
Nov 03 04:30:27.259905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:27.259853    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" podStartSLOduration=2.259839897 podStartE2EDuration="2.259839897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:30:27.258960547 +0000 UTC m=+5318.489852964" watchObservedRunningTime="2026-11-03 04:30:27.259839897 +0000 UTC m=+5318.490732315"
Nov 03 04:30:28.004277 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:28.004245    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:36.004139 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:36.004096    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:36.016739 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:36.016715    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:36.194370 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:36.194343    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:30:37.293495 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:37.293452    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:30:46.881179 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:30:46.881141    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:30:50.983644 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:30:50.983603    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:31:13.466543 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:13.466506    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:31:18.895744 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:18.895704    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:31:20.983488 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:31:20.983447    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:31:21.342101 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:21.342000    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:31:40.996567 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:40.996517    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:31:40.997818 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:40.997795    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"]
Nov 03 04:31:41.027629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:41.027604    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:31:41.175995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:41.175966    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:31:48.785711 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:48.785646    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:31:48.786429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:48.786385    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:31:48.851693 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:48.851660    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:31:48.851862 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:48.851707    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:31:48.904681 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:48.904641    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:31:48.904991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:48.904958    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:31:50.448314 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:50.448289    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:31:50.450305 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:50.450284    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:31:50.508937 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:50.508911    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:31:50.511152 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:50.511132    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:31:50.983065 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:31:50.983019    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:31:53.068337 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:53.068261    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:31:55.646050 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:55.646022    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:31:55.852692 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:55.852650    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:31:56.018736 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.018640    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" podUID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" containerName="main" containerID="cri-o://a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44" gracePeriod=45
Nov 03 04:31:56.184378 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.184337    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" podUID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.232:8000/ready\": dial tcp 10.134.0.232:8000: connect: connection refused"
Nov 03 04:31:56.282739 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.282672    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:31:56.284853 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.284834    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd\" (UniqueName: \"kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd\") pod \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") "
Nov 03 04:31:56.284969 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.284871    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home\") pod \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") "
Nov 03 04:31:56.284969 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.284913    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs\") pod \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") "
Nov 03 04:31:56.285088 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.284973    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm\") pod \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") "
Nov 03 04:31:56.285088 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.285037    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache\") pod \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") "
Nov 03 04:31:56.285207 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.285100    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir\") pod \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\" (UID: \"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e\") "
Nov 03 04:31:56.288581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.285489    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache" pod "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" (UID: "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:31:56.288581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.285215    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home" pod "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" (UID: "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:31:56.288581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.285830    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir" pod "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" (UID: "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:31:56.288581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.287457    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs" pod "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" (UID: "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:31:56.288581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.287649    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd" pod "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" (UID: "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e"). InnerVolumeSpecName "kube-api-access-6x8hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:31:56.288581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.287839    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm" pod "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" (UID: "651fd98c-04d6-4d7d-8cbd-a7551ea06e1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:31:56.386308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.386273    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:31:56.386308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.386304    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:31:56.386308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.386315    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:31:56.386591 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.386324    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8hd\" (UniqueName: \"kubernetes.io/projected/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-kube-api-access-6x8hd\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:31:56.386591 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.386334    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:31:56.386591 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.386342    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:31:56.598804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.598721    5515 generic.go:356] "Generic (PLEG): container finished" podID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" containerID="a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44" exitCode=0
Nov 03 04:31:56.598804 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.598777    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"
Nov 03 04:31:56.599000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.598781    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" event={"ID":"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e","Type":"ContainerDied","Data":"a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44"}
Nov 03 04:31:56.599000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.598886    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz" event={"ID":"651fd98c-04d6-4d7d-8cbd-a7551ea06e1e","Type":"ContainerDied","Data":"32acae74176ade69718457b1897331265d96425ad3bea48921028f1330e7d8fc"}
Nov 03 04:31:56.599000 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.598903    5515 scope.go:122] "RemoveContainer" containerID="a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44"
Nov 03 04:31:56.608713 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.608696    5515 scope.go:122] "RemoveContainer" containerID="a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44"
Nov 03 04:31:56.608993 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:31:56.608973    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44\": container with ID starting with a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44 not found: ID does not exist" containerID="a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44"
Nov 03 04:31:56.609061 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.609002    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44"} err="failed to get container status \"a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44\": rpc error: code = NotFound desc = could not find container \"a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44\": container with ID starting with a49da7497d40e3405f8d8fa219d2983f6eb538723589e48be3390cef5bd0cb44 not found: ID does not exist"
Nov 03 04:31:56.765903 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.765866    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"]
Nov 03 04:31:56.768313 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.768290    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-edfabb23-kserve-7644fcd5fb-2nbdz"]
Nov 03 04:31:56.893185 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:56.893147    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" path="/var/lib/kubelet/pods/651fd98c-04d6-4d7d-8cbd-a7551ea06e1e/volumes"
Nov 03 04:31:59.461850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:31:59.461816    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:00.037947 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.037916    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"]
Nov 03 04:32:00.038326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.038314    5515 cpu_manager.go:414] "containerMap: removing container" podUID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" containerName="main"
Nov 03 04:32:00.038326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.038327    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" containerName="main"
Nov 03 04:32:00.038438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.038423    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="651fd98c-04d6-4d7d-8cbd-a7551ea06e1e" containerName="main"
Nov 03 04:32:00.041454 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.041428    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.097199 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.097170    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-deca42-openshift-default-dockercfg-mnphm\""
Nov 03 04:32:00.127965 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.127933    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.127994    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128081    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128116    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128132    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128152    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128357 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128237    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128357 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128336    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.128428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.128398    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chb6m\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.147099 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.147070    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:00.153020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.152986    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"]
Nov 03 04:32:00.229822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.229789    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.229835    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.229871    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chb6m\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.229916    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.229959    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.229982    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230012    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230041 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230028    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230053    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230322    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230356    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230419 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230407    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230726 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230704    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.230850 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.230831    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.232441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.232419    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.232713 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.232697    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.292538 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.292470    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chb6m\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.293852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.293828    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token\") pod \"e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") " pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.351824 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.351787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:00.353742 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.353721    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:00.778936 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.778909    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"]
Nov 03 04:32:00.780818 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:32:00.780789    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8badc312_6536_4dab_bec9_1843688e8bf5.slice/crio-6c867526c806353e3197c1ed4e4aa4b3482b479aedd301de2f1109e1c18a015e WatchSource:0}: Error finding container 6c867526c806353e3197c1ed4e4aa4b3482b479aedd301de2f1109e1c18a015e: Status 404 returned error can't find the container with id 6c867526c806353e3197c1ed4e4aa4b3482b479aedd301de2f1109e1c18a015e
Nov 03 04:32:00.782977 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.782945    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:32:00.783060 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.783012    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:32:00.783060 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:00.783042    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:32:01.624096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:01.624056    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" event={"ID":"8badc312-6536-4dab-bec9-1843688e8bf5","Type":"ContainerStarted","Data":"20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"}
Nov 03 04:32:01.624311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:01.624105    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" event={"ID":"8badc312-6536-4dab-bec9-1843688e8bf5","Type":"ContainerStarted","Data":"6c867526c806353e3197c1ed4e4aa4b3482b479aedd301de2f1109e1c18a015e"}
Nov 03 04:32:01.799184 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:01.799129    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podStartSLOduration=2.79911573 podStartE2EDuration="2.79911573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:32:01.797532226 +0000 UTC m=+5413.028424645" watchObservedRunningTime="2026-11-03 04:32:01.79911573 +0000 UTC m=+5413.030008147"
Nov 03 04:32:02.354509 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:02.354470    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:02.359577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:02.359533    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:02.629986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:02.629958    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:02.630801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:02.630783    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:04.810704 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:04.810601    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:04.993464 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:04.993373    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:05.169039 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:05.168961    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:08.099437 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.099399    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:08.282428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.282379    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:08.536640 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.536608    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"]
Nov 03 04:32:08.540152 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.540131    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.598416 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.598379    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"]
Nov 03 04:32:08.655321 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.655287    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-deca42\""
Nov 03 04:32:08.672796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.671446    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7jq\" (UniqueName: \"kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq\") pod \"payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.672796 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.671567    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.688252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.685789    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"]
Nov 03 04:32:08.690115 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.690087    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:08.732188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.732156    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"]
Nov 03 04:32:08.743616 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.743591    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8796a67c-dockercfg-s92x4\""
Nov 03 04:32:08.772725 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.772692    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7jq\" (UniqueName: \"kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq\") pod \"payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.772909 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.772767    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.773481 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.773456    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.835409 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.835329    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7jq\" (UniqueName: \"kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq\") pod \"payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.852893 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.852866    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:08.873817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.873788    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jx4\" (UniqueName: \"kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4\") pod \"payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:08.873949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.873829    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:08.974621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.974582    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jx4\" (UniqueName: \"kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4\") pod \"payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:08.974621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.974623    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:08.975367 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:08.975345    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:09.037423 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.037386    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jx4\" (UniqueName: \"kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4\") pod \"payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:09.227999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.227974    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"]
Nov 03 04:32:09.230144 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:32:09.230112    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06b1529_454c_4afc_b3cf_33a5abfc6e5d.slice/crio-09e17d86bc2802f44fed31bacb2ba2ea54c556672eb21f59a0ba197e45a5ad5f WatchSource:0}: Error finding container 09e17d86bc2802f44fed31bacb2ba2ea54c556672eb21f59a0ba197e45a5ad5f: Status 404 returned error can't find the container with id 09e17d86bc2802f44fed31bacb2ba2ea54c556672eb21f59a0ba197e45a5ad5f
Nov 03 04:32:09.304918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.304893    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:09.678599 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:32:09.678120    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00aaac03_9ee7_4d5c_b4ec_835c9758bd3d.slice/crio-9166d99ee8bc138c40586b453656b9d25e289e23a5ada165c18a6d0b3a597faf WatchSource:0}: Error finding container 9166d99ee8bc138c40586b453656b9d25e289e23a5ada165c18a6d0b3a597faf: Status 404 returned error can't find the container with id 9166d99ee8bc138c40586b453656b9d25e289e23a5ada165c18a6d0b3a597faf
Nov 03 04:32:09.684671 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.682331    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" event={"ID":"f06b1529-454c-4afc-b3cf-33a5abfc6e5d","Type":"ContainerStarted","Data":"5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9"}
Nov 03 04:32:09.684671 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.682434    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" event={"ID":"f06b1529-454c-4afc-b3cf-33a5abfc6e5d","Type":"ContainerStarted","Data":"09e17d86bc2802f44fed31bacb2ba2ea54c556672eb21f59a0ba197e45a5ad5f"}
Nov 03 04:32:09.684671 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.684007    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:09.686667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.686640    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"]
Nov 03 04:32:09.801518 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:09.801461    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" podStartSLOduration=1.8014473579999999 podStartE2EDuration="1.801447358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:32:09.800952252 +0000 UTC m=+5421.031844670" watchObservedRunningTime="2026-11-03 04:32:09.801447358 +0000 UTC m=+5421.032339775"
Nov 03 04:32:10.688909 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:10.688871    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" event={"ID":"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d","Type":"ContainerStarted","Data":"a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185"}
Nov 03 04:32:10.689399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:10.689113    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:10.689399 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:10.689148    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" event={"ID":"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d","Type":"ContainerStarted","Data":"9166d99ee8bc138c40586b453656b9d25e289e23a5ada165c18a6d0b3a597faf"}
Nov 03 04:32:10.807481 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:10.807429    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" podStartSLOduration=2.807416237 podStartE2EDuration="2.807416237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:32:10.806590912 +0000 UTC m=+5422.037483326" watchObservedRunningTime="2026-11-03 04:32:10.807416237 +0000 UTC m=+5422.038308655"
Nov 03 04:32:11.043933 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.043840    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"]
Nov 03 04:32:11.047452 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.047431    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.101190 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.101157    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-deca42-serving-cert\""
Nov 03 04:32:11.107889 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.107858    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"]
Nov 03 04:32:11.196958 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.196918    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.197138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.197001    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.197138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.197113    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.197222 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.197160    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grz7\" (UniqueName: \"kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.242746 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.242705    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:11.298073 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.297989    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.298073 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.298028    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grz7\" (UniqueName: \"kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.298319 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.298089    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.298319 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.298124    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.298801 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.298777    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.300777 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.300760    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.300843 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.300784    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.361173 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.361144    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grz7\" (UniqueName: \"kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.361341 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.361315    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:11.402392 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.402362    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:11.549859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.549781    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:11.736301 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.736275    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"]
Nov 03 04:32:11.738064 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:32:11.738031    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10113da9_2e4b_47b1_a62b_5eb2d96cae8e.slice/crio-4123f73145e9e44bd3720be8de9541f2968aa53eb7af11c19f8ac7d59aaf00fa WatchSource:0}: Error finding container 4123f73145e9e44bd3720be8de9541f2968aa53eb7af11c19f8ac7d59aaf00fa: Status 404 returned error can't find the container with id 4123f73145e9e44bd3720be8de9541f2968aa53eb7af11c19f8ac7d59aaf00fa
Nov 03 04:32:11.739560 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:11.739529    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:32:12.704932 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:12.704895    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" event={"ID":"10113da9-2e4b-47b1-a62b-5eb2d96cae8e","Type":"ContainerStarted","Data":"b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8"}
Nov 03 04:32:12.705171 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:12.704947    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" event={"ID":"10113da9-2e4b-47b1-a62b-5eb2d96cae8e","Type":"ContainerStarted","Data":"4123f73145e9e44bd3720be8de9541f2968aa53eb7af11c19f8ac7d59aaf00fa"}
Nov 03 04:32:12.705171 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:12.704990    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:12.846784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:12.846731    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" podStartSLOduration=0.970769627 podStartE2EDuration="1.846715816s" totalImagesPullingTime="875.946189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:32:12.84597428 +0000 UTC m=+5424.076866697" watchObservedRunningTime="2026-11-03 04:32:12.846715816 +0000 UTC m=+5424.077608246"
Nov 03 04:32:16.187297 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:16.187247    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:18.715817 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:18.715788    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:32:19.245761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:19.245720    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:19.466347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:19.466318    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:32:21.005845 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:32:21.005805    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:32:21.186578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:21.184337    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:21.697067 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:21.697038    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:32:21.697308 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:21.697114    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:32:25.994238 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:25.994190    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:28.813433 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:28.813343    5515 generic.go:356] "Generic (PLEG): container finished" podID="8badc312-6536-4dab-bec9-1843688e8bf5" containerID="20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72" exitCode=0
Nov 03 04:32:28.814010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:28.813426    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" event={"ID":"8badc312-6536-4dab-bec9-1843688e8bf5","Type":"ContainerDied","Data":"20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"}
Nov 03 04:32:28.814010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:28.813905    5515 scope.go:122] "RemoveContainer" containerID="20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"
Nov 03 04:32:28.815176 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:28.815141    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:32:28.815302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:28.815209    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:32:28.815302 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:28.815260    5515 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 04:32:29.682573 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:29.679055    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:29.820174 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:29.820141    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" event={"ID":"8badc312-6536-4dab-bec9-1843688e8bf5","Type":"ContainerStarted","Data":"39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe"}
Nov 03 04:32:30.312654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:30.312623    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:30.354861 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:30.354829    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:30.355985 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:30.355962    5515 patch_prober.go:29] interesting pod/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 03 04:32:30.356044 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:30.356012    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 03 04:32:30.460821 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:30.460789    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:30.611533 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:30.611504    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:31.138942 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:31.138916    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:31.354890 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:31.354854    5515 patch_prober.go:29] interesting pod/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 03 04:32:31.355073 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:31.354913    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 03 04:32:32.354728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:32.354686    5515 patch_prober.go:29] interesting pod/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 03 04:32:32.355152 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:32.354756    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 03 04:32:32.630582 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:32.630530    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:33.354188 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:33.354155    5515 patch_prober.go:29] interesting pod/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 03 04:32:33.354373 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:33.354221    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 03 04:32:34.354223 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:34.354189    5515 patch_prober.go:29] interesting pod/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Nov 03 04:32:34.354677 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:34.354256    5515 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Nov 03 04:32:34.427029 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:34.426997    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:35.358416 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:35.358381    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:35.359335 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:35.359319    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:32:35.977577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:35.977516    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:39.209175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:39.209134    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:32:40.782398 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:40.782364    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:45.487943 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:45.487881    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:50.295513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:50.295475    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:50.984122 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:32:50.984082    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:32:55.081226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:55.081185    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:32:59.922869 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:32:59.922839    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:03.904842 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:03.904802    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:33:04.698777 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:04.698744    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:09.492040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:09.491993    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:14.286445 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:14.286406    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:19.162686 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:19.162640    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:19.586231 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:19.586128    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:20.984103 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:33:20.984067    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:33:22.240945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:22.240909    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:23.887742 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:23.887709    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:28.785912 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:28.785882    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:30.167470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:30.167421    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:30.680013 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:30.679979    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:32.087379 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:32.087342    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:33:33.797037 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:33.797005    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:35.326348 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:35.326284    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:37.199029 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:37.198998    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:37.551802 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:37.551718    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:37.873835 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:37.873799    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"]
Nov 03 04:33:37.995055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:37.995019    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"]
Nov 03 04:33:38.170828 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:38.170747    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"]
Nov 03 04:33:38.170999 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:38.170957    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" podUID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" containerName="maas-api" containerID="cri-o://b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8" gracePeriod=30
Nov 03 04:33:38.389118 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:38.389087    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:38.672074 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:38.672026    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:38.710755 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:38.710705    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" podUID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.236:8443/health\": dial tcp 10.134.0.236:8443: connect: connection refused"
Nov 03 04:33:39.467912 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.467883    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:33:39.622097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.622066    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7\" (UniqueName: \"kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7\") pod \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") "
Nov 03 04:33:39.622311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.622156    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls\") pod \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") "
Nov 03 04:33:39.622311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.622224    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca\") pod \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") "
Nov 03 04:33:39.622311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.622292    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs\") pod \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\" (UID: \"10113da9-2e4b-47b1-a62b-5eb2d96cae8e\") "
Nov 03 04:33:39.622668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.622633    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca" pod "10113da9-2e4b-47b1-a62b-5eb2d96cae8e" (UID: "10113da9-2e4b-47b1-a62b-5eb2d96cae8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:33:39.625079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.625032    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls" pod "10113da9-2e4b-47b1-a62b-5eb2d96cae8e" (UID: "10113da9-2e4b-47b1-a62b-5eb2d96cae8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:33:39.625230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.625126    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs" pod "10113da9-2e4b-47b1-a62b-5eb2d96cae8e" (UID: "10113da9-2e4b-47b1-a62b-5eb2d96cae8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:33:39.625230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.625147    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7" pod "10113da9-2e4b-47b1-a62b-5eb2d96cae8e" (UID: "10113da9-2e4b-47b1-a62b-5eb2d96cae8e"). InnerVolumeSpecName "kube-api-access-4grz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:33:39.724063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.724023    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:33:39.724063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.724062    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:33:39.724063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.724071    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:33:39.724347 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:39.724082    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grz7\" (UniqueName: \"kubernetes.io/projected/10113da9-2e4b-47b1-a62b-5eb2d96cae8e-kube-api-access-4grz7\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:33:40.185534 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.185496    5515 generic.go:356] "Generic (PLEG): container finished" podID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" containerID="b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8" exitCode=0
Nov 03 04:33:40.185741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.185558    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"
Nov 03 04:33:40.185741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.185583    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" event={"ID":"10113da9-2e4b-47b1-a62b-5eb2d96cae8e","Type":"ContainerDied","Data":"b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8"}
Nov 03 04:33:40.185741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.185646    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf" event={"ID":"10113da9-2e4b-47b1-a62b-5eb2d96cae8e","Type":"ContainerDied","Data":"4123f73145e9e44bd3720be8de9541f2968aa53eb7af11c19f8ac7d59aaf00fa"}
Nov 03 04:33:40.185741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.185676    5515 scope.go:122] "RemoveContainer" containerID="b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8"
Nov 03 04:33:40.195972 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.195949    5515 scope.go:122] "RemoveContainer" containerID="b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8"
Nov 03 04:33:40.196308 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:33:40.196285    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8\": container with ID starting with b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8 not found: ID does not exist" containerID="b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8"
Nov 03 04:33:40.196355 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.196318    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8"} err="failed to get container status \"b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8\": rpc error: code = NotFound desc = could not find container \"b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8\": container with ID starting with b104807e0d11b1fb4c9499a281ae176252bb0cf759cb078b0743943b664f5aa8 not found: ID does not exist"
Nov 03 04:33:40.367462 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.367426    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"]
Nov 03 04:33:40.418204 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.418173    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-55knf"]
Nov 03 04:33:40.892537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:40.892502    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" path="/var/lib/kubelet/pods/10113da9-2e4b-47b1-a62b-5eb2d96cae8e/volumes"
Nov 03 04:33:42.683175 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:42.683135    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:33:42.874174 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:42.874128    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerName="payload-processing" containerID="cri-o://a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185" gracePeriod=25
Nov 03 04:33:42.996048 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:42.995956    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerName="payload-pre-processing" containerID="cri-o://5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9" gracePeriod=25
Nov 03 04:33:43.581172 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:43.581115    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:43.738953 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:43.738918    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:33:50.984975 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:33:50.984938    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:33:51.589960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:51.589927    5515 scope.go:122] "RemoveContainer" containerID="856334ed09a4ccb1b7d1123951dba06756a4042ea4da16a21cabd3a8f6291fe2"
Nov 03 04:33:51.695721 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:51.695675    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:33:51.695924 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:33:51.695751    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:34:01.695460 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:01.695421    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:34:01.695942 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:01.695503    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:34:03.149831 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.149801    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:34:03.251108 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.251082    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume\") pod \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") "
Nov 03 04:34:03.251242 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.251134    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4\" (UniqueName: \"kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4\") pod \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\" (UID: \"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d\") "
Nov 03 04:34:03.251485 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.251460    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume" pod "00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" (UID: "00aaac03-9ee7-4d5c-b4ec-835c9758bd3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:34:03.253477 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.253455    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4" pod "00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" (UID: "00aaac03-9ee7-4d5c-b4ec-835c9758bd3d"). InnerVolumeSpecName "kube-api-access-m6jx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:34:03.255826 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.255808    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:34:03.297207 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.297179    5515 generic.go:356] "Generic (PLEG): container finished" podID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerID="a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185" exitCode=0
Nov 03 04:34:03.297362 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.297227    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"
Nov 03 04:34:03.297362 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.297270    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" event={"ID":"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d","Type":"ContainerDied","Data":"a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185"}
Nov 03 04:34:03.297683 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.297391    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv" event={"ID":"00aaac03-9ee7-4d5c-b4ec-835c9758bd3d","Type":"ContainerDied","Data":"9166d99ee8bc138c40586b453656b9d25e289e23a5ada165c18a6d0b3a597faf"}
Nov 03 04:34:03.297683 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.297427    5515 scope.go:122] "RemoveContainer" containerID="a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185"
Nov 03 04:34:03.299087 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.299047    5515 generic.go:356] "Generic (PLEG): container finished" podID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerID="5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9" exitCode=0
Nov 03 04:34:03.299170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.299089    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"
Nov 03 04:34:03.299170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.299150    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" event={"ID":"f06b1529-454c-4afc-b3cf-33a5abfc6e5d","Type":"ContainerDied","Data":"5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9"}
Nov 03 04:34:03.299666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.299189    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x" event={"ID":"f06b1529-454c-4afc-b3cf-33a5abfc6e5d","Type":"ContainerDied","Data":"09e17d86bc2802f44fed31bacb2ba2ea54c556672eb21f59a0ba197e45a5ad5f"}
Nov 03 04:34:03.311665 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.311647    5515 scope.go:122] "RemoveContainer" containerID="a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185"
Nov 03 04:34:03.311994 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:03.311967    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185\": container with ID starting with a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185 not found: ID does not exist" containerID="a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185"
Nov 03 04:34:03.312122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.311998    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185"} err="failed to get container status \"a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185\": rpc error: code = NotFound desc = could not find container \"a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185\": container with ID starting with a246b52e27edfef933a7755902e0e4d7bf50760f5879e4dff4770dc0ba5c2185 not found: ID does not exist"
Nov 03 04:34:03.312122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.312015    5515 scope.go:122] "RemoveContainer" containerID="5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9"
Nov 03 04:34:03.319666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.319649    5515 scope.go:122] "RemoveContainer" containerID="5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9"
Nov 03 04:34:03.319950 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:03.319932    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9\": container with ID starting with 5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9 not found: ID does not exist" containerID="5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9"
Nov 03 04:34:03.319998 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.319957    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9"} err="failed to get container status \"5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9\": rpc error: code = NotFound desc = could not find container \"5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9\": container with ID starting with 5355c659a3c12b15190cc5ce43613af585337a3ebbcfaf18d7b0e502dbd413e9 not found: ID does not exist"
Nov 03 04:34:03.352031 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.352010    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq\" (UniqueName: \"kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq\") pod \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") "
Nov 03 04:34:03.352113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.352050    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume\") pod \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\" (UID: \"f06b1529-454c-4afc-b3cf-33a5abfc6e5d\") "
Nov 03 04:34:03.352283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.352271    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:03.352328 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.352286    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jx4\" (UniqueName: \"kubernetes.io/projected/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d-kube-api-access-m6jx4\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:03.352441 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.352423    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume" pod "f06b1529-454c-4afc-b3cf-33a5abfc6e5d" (UID: "f06b1529-454c-4afc-b3cf-33a5abfc6e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:34:03.354073 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.354054    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq" pod "f06b1529-454c-4afc-b3cf-33a5abfc6e5d" (UID: "f06b1529-454c-4afc-b3cf-33a5abfc6e5d"). InnerVolumeSpecName "kube-api-access-px7jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:34:03.453637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.453539    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7jq\" (UniqueName: \"kubernetes.io/projected/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-kube-api-access-px7jq\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:03.453637 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.453590    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b1529-454c-4afc-b3cf-33a5abfc6e5d-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:03.563324 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.563289    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"]
Nov 03 04:34:03.566418 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.566387    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-deca42-5fb4f4ff8qmdxv"]
Nov 03 04:34:03.722832 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.722752    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"]
Nov 03 04:34:03.729282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:03.729259    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-deca42-66f5fcrx8x"]
Nov 03 04:34:04.892614 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:04.892579    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" path="/var/lib/kubelet/pods/00aaac03-9ee7-4d5c-b4ec-835c9758bd3d/volumes"
Nov 03 04:34:04.893098 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:04.892986    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" path="/var/lib/kubelet/pods/f06b1529-454c-4afc-b3cf-33a5abfc6e5d/volumes"
Nov 03 04:34:12.592325 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:12.592285    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:12.757333 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:12.757301    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:13.288644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:13.288610    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:14.937007 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.936978    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"]
Nov 03 04:34:14.937422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937406    5515 cpu_manager.go:414] "containerMap: removing container" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerName="payload-pre-processing"
Nov 03 04:34:14.937468 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937423    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerName="payload-pre-processing"
Nov 03 04:34:14.937468 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937457    5515 cpu_manager.go:414] "containerMap: removing container" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerName="payload-processing"
Nov 03 04:34:14.937468 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937463    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerName="payload-processing"
Nov 03 04:34:14.937592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937478    5515 cpu_manager.go:414] "containerMap: removing container" podUID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" containerName="maas-api"
Nov 03 04:34:14.937592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937483    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" containerName="maas-api"
Nov 03 04:34:14.937592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937561    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="00aaac03-9ee7-4d5c-b4ec-835c9758bd3d" containerName="payload-processing"
Nov 03 04:34:14.937592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937571    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="10113da9-2e4b-47b1-a62b-5eb2d96cae8e" containerName="maas-api"
Nov 03 04:34:14.937592 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.937578    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06b1529-454c-4afc-b3cf-33a5abfc6e5d" containerName="payload-pre-processing"
Nov 03 04:34:14.941792 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.941775    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:14.995282 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:14.995246    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-deca42-serving-cert\""
Nov 03 04:34:15.047978 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.047943    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"]
Nov 03 04:34:15.076792 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.076757    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9nl\" (UniqueName: \"kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.076969 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.076821    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.076969 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.076944    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.077074 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.077059    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.145519 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.145483    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:15.190002 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.189916    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.195945 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.195596    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9nl\" (UniqueName: \"kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.196300 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.196261    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.196578 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.196532    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.197497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.195441    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.199765 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:15.199205    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-serving-cert: secret "maas-api-e2e-worker-single-main-deca42-serving-cert" not found
Nov 03 04:34:15.199898 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:15.199822    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls podName:21912c46-72dd-4488-9c21-e7f567b75892 nodeName:}" failed. No retries permitted until 2026-11-03 04:34:15.699797965 +0000 UTC m=+5546.930690377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls") pod "maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" (UID: "21912c46-72dd-4488-9c21-e7f567b75892") : secret "maas-api-e2e-worker-single-main-deca42-serving-cert" not found
Nov 03 04:34:15.203618 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.203588    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.259451 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.259421    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9nl\" (UniqueName: \"kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.294610 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.294577    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:15.443880 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.443797    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:15.703372 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.703286    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.705927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.705903    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:15.861372 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:15.861342    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:16.236532 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:16.236504    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"]
Nov 03 04:34:16.238597 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:34:16.238567    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21912c46_72dd_4488_9c21_e7f567b75892.slice/crio-e2719f3ac7b4f87298c94065bd26876f6f7e829e72a485df0cfaff3b7a4ff421 WatchSource:0}: Error finding container e2719f3ac7b4f87298c94065bd26876f6f7e829e72a485df0cfaff3b7a4ff421: Status 404 returned error can't find the container with id e2719f3ac7b4f87298c94065bd26876f6f7e829e72a485df0cfaff3b7a4ff421
Nov 03 04:34:16.363995 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:16.363961    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" event={"ID":"21912c46-72dd-4488-9c21-e7f567b75892","Type":"ContainerStarted","Data":"e2719f3ac7b4f87298c94065bd26876f6f7e829e72a485df0cfaff3b7a4ff421"}
Nov 03 04:34:17.370730 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:17.370698    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" event={"ID":"21912c46-72dd-4488-9c21-e7f567b75892","Type":"ContainerStarted","Data":"3b2d991142513ce27a35fc0aafa407fef739bb35bd042fc98c00872769237ff9"}
Nov 03 04:34:17.371090 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:17.370797    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:17.493872 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:17.493780    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" podStartSLOduration=2.552810155 podStartE2EDuration="3.493765832s" totalImagesPullingTime="940.955677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:34:17.492640141 +0000 UTC m=+5548.723532562" watchObservedRunningTime="2026-11-03 04:34:17.493765832 +0000 UTC m=+5548.724658269"
Nov 03 04:34:19.429235 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:19.429204    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:19.584821 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:19.584787    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:19.946732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:19.946704    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:20.105800 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:20.105762    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:20.987435 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:20.987396    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:34:23.381852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:23.381821    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:23.406375 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:23.406349    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:34:24.894354 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:24.894317    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:26.053666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:26.053638    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:29.593980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:29.593946    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:34.468438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:34.468401    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:39.178468 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:39.178436    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:43.303033 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.302992    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"]
Nov 03 04:34:43.303443 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.303242    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" podUID="21912c46-72dd-4488-9c21-e7f567b75892" containerName="maas-api" containerID="cri-o://3b2d991142513ce27a35fc0aafa407fef739bb35bd042fc98c00872769237ff9" gracePeriod=30
Nov 03 04:34:43.377281 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.377246    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" podUID="21912c46-72dd-4488-9c21-e7f567b75892" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.237:8443/health\": dial tcp 10.134.0.237:8443: connect: connection refused"
Nov 03 04:34:43.423948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.423918    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:43.495316 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.495286    5515 generic.go:356] "Generic (PLEG): container finished" podID="21912c46-72dd-4488-9c21-e7f567b75892" containerID="3b2d991142513ce27a35fc0aafa407fef739bb35bd042fc98c00872769237ff9" exitCode=0
Nov 03 04:34:43.495475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.495368    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" event={"ID":"21912c46-72dd-4488-9c21-e7f567b75892","Type":"ContainerDied","Data":"3b2d991142513ce27a35fc0aafa407fef739bb35bd042fc98c00872769237ff9"}
Nov 03 04:34:43.579029 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.579004    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:43.672055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.672024    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca\") pod \"21912c46-72dd-4488-9c21-e7f567b75892\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") "
Nov 03 04:34:43.672226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.672085    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl\" (UniqueName: \"kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl\") pod \"21912c46-72dd-4488-9c21-e7f567b75892\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") "
Nov 03 04:34:43.672226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.672113    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs\") pod \"21912c46-72dd-4488-9c21-e7f567b75892\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") "
Nov 03 04:34:43.672226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.672148    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\") pod \"21912c46-72dd-4488-9c21-e7f567b75892\" (UID: \"21912c46-72dd-4488-9c21-e7f567b75892\") "
Nov 03 04:34:43.672454 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.672431    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca" pod "21912c46-72dd-4488-9c21-e7f567b75892" (UID: "21912c46-72dd-4488-9c21-e7f567b75892"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:34:43.674436 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.674409    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls" pod "21912c46-72dd-4488-9c21-e7f567b75892" (UID: "21912c46-72dd-4488-9c21-e7f567b75892"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:34:43.674576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.674472    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs" pod "21912c46-72dd-4488-9c21-e7f567b75892" (UID: "21912c46-72dd-4488-9c21-e7f567b75892"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:34:43.674644 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.674627    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl" pod "21912c46-72dd-4488-9c21-e7f567b75892" (UID: "21912c46-72dd-4488-9c21-e7f567b75892"). InnerVolumeSpecName "kube-api-access-bq9nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:34:43.773303 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.773271    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:43.773303 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.773299    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21912c46-72dd-4488-9c21-e7f567b75892-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:43.773303 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.773308    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9nl\" (UniqueName: \"kubernetes.io/projected/21912c46-72dd-4488-9c21-e7f567b75892-kube-api-access-bq9nl\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:43.773539 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:43.773317    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21912c46-72dd-4488-9c21-e7f567b75892-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:44.501006 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.500975    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"
Nov 03 04:34:44.501432 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.501011    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv" event={"ID":"21912c46-72dd-4488-9c21-e7f567b75892","Type":"ContainerDied","Data":"e2719f3ac7b4f87298c94065bd26876f6f7e829e72a485df0cfaff3b7a4ff421"}
Nov 03 04:34:44.501432 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.501074    5515 scope.go:122] "RemoveContainer" containerID="3b2d991142513ce27a35fc0aafa407fef739bb35bd042fc98c00872769237ff9"
Nov 03 04:34:44.668477 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.668444    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"]
Nov 03 04:34:44.719306 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.719279    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-deca42-57dfbf8dfd-q7nfv"]
Nov 03 04:34:44.893475 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.893433    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21912c46-72dd-4488-9c21-e7f567b75892" path="/var/lib/kubelet/pods/21912c46-72dd-4488-9c21-e7f567b75892/volumes"
Nov 03 04:34:44.971095 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:44.971017    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:45.241471 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:45.241391    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:45.891793 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:45.891762    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"]
Nov 03 04:34:45.892252 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:45.891965    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" containerID="cri-o://39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe" gracePeriod=30
Nov 03 04:34:45.912742 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:45.912719    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:49.975594 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:49.975540    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:34:50.360751 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:50.360720    5515 patch_prober.go:29] interesting pod/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 04:34:50.360937 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:50.360788    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 04:34:50.983927 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:50.983878    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:34:51.143096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.143067    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:34:51.252336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252259    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252308    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252336 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252331    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252367    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252427    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252472    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252499    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252575    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252606    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket\") pod \"8badc312-6536-4dab-bec9-1843688e8bf5\" (UID: \"8badc312-6536-4dab-bec9-1843688e8bf5\") "
Nov 03 04:34:51.252667 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252612    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:34:51.253075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252979    5515 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.253075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252983    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:34:51.253075 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.252995    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:34:51.253406 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.253067    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:34:51.253406 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.253354    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:34:51.254770 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.254744    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:34:51.254902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.254814    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:34:51.254902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.254832    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "kube-api-access-chb6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:34:51.254982 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.254942    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo" pod "8badc312-6536-4dab-bec9-1843688e8bf5" (UID: "8badc312-6536-4dab-bec9-1843688e8bf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 04:34:51.354283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354248    5515 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8badc312-6536-4dab-bec9-1843688e8bf5-istio-podinfo\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354278    5515 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8badc312-6536-4dab-bec9-1843688e8bf5-istiod-ca-cert\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354287    5515 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-data\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354296    5515 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-credential-socket\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354306    5515 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-istio-token\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354313    5515 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-istio-envoy\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354322    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chb6m\" (UniqueName: \"kubernetes.io/projected/8badc312-6536-4dab-bec9-1843688e8bf5-kube-api-access-chb6m\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.354523 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.354332    5515 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8badc312-6536-4dab-bec9-1843688e8bf5-workload-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:34:51.537609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.537495    5515 generic.go:356] "Generic (PLEG): container finished" podID="8badc312-6536-4dab-bec9-1843688e8bf5" containerID="39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe" exitCode=0
Nov 03 04:34:51.537609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.537578    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"
Nov 03 04:34:51.537609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.537589    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" event={"ID":"8badc312-6536-4dab-bec9-1843688e8bf5","Type":"ContainerDied","Data":"39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe"}
Nov 03 04:34:51.537894 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.537658    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj" event={"ID":"8badc312-6536-4dab-bec9-1843688e8bf5","Type":"ContainerDied","Data":"6c867526c806353e3197c1ed4e4aa4b3482b479aedd301de2f1109e1c18a015e"}
Nov 03 04:34:51.537894 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.537688    5515 scope.go:122] "RemoveContainer" containerID="39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe"
Nov 03 04:34:51.547215 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.547197    5515 scope.go:122] "RemoveContainer" containerID="20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"
Nov 03 04:34:51.558069 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.558053    5515 scope.go:122] "RemoveContainer" containerID="39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe"
Nov 03 04:34:51.558338 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:51.558317    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe\": container with ID starting with 39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe not found: ID does not exist" containerID="39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe"
Nov 03 04:34:51.558429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.558350    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe"} err="failed to get container status \"39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe\": rpc error: code = NotFound desc = could not find container \"39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe\": container with ID starting with 39bbc7f9795e30a6515763f7abe17cdf23e436419b6e59a52376d5d850af8afe not found: ID does not exist"
Nov 03 04:34:51.558429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.558374    5515 scope.go:122] "RemoveContainer" containerID="20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"
Nov 03 04:34:51.558585 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:34:51.558542    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72\": container with ID starting with 20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72 not found: ID does not exist" containerID="20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"
Nov 03 04:34:51.558649 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.558601    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72"} err="failed to get container status \"20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72\": rpc error: code = NotFound desc = could not find container \"20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72\": container with ID starting with 20d7105dfb615c9f7b87055dd54e3ac12b93e7153cb8108db3e289f3c9cb8a72 not found: ID does not exist"
Nov 03 04:34:51.715285 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.715256    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"]
Nov 03 04:34:51.716437 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:51.716419    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-deca42-openshift-default-6f4c965857t74sj"]
Nov 03 04:34:52.892180 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:52.892103    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" path="/var/lib/kubelet/pods/8badc312-6536-4dab-bec9-1843688e8bf5/volumes"
Nov 03 04:34:55.125563 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.125517    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"]
Nov 03 04:34:55.126038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.125904    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy"
Nov 03 04:34:55.126038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.125914    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy"
Nov 03 04:34:55.126038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.125932    5515 cpu_manager.go:414] "containerMap: removing container" podUID="21912c46-72dd-4488-9c21-e7f567b75892" containerName="maas-api"
Nov 03 04:34:55.126038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.125937    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21912c46-72dd-4488-9c21-e7f567b75892" containerName="maas-api"
Nov 03 04:34:55.126038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.126006    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy"
Nov 03 04:34:55.126038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.126021    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="21912c46-72dd-4488-9c21-e7f567b75892" containerName="maas-api"
Nov 03 04:34:55.130629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.130610    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.209221 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.209167    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:34:55.232720 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.232699    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-82587b78-kserve-self-signed-certs\""
Nov 03 04:34:55.238960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.238939    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"]
Nov 03 04:34:55.289593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.289530    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.289766 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.289646    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.289766 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.289726    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2hc\" (UniqueName: \"kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.289861 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.289783    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.289861 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.289816    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.289978 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.289951    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390679 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390608    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390679 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390652    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390777    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390852    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390879    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390927    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2hc\" (UniqueName: \"kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.390980 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.390963    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.391217 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.391199    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.391277 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.391265    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.393183 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.393151    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.393299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.393244    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.452920 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.452892    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2hc\" (UniqueName: \"kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc\") pod \"e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") " pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:55.741912 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:55.741833    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:34:56.131724 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:56.131698    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"]
Nov 03 04:34:56.133201 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:34:56.133173    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a91a2b8_3399_4b58_8cba_2ceebb206ae5.slice/crio-be19a4ad89cbde2e0febdd8e218ac9be219daa5a2e8c032999d8f7494ca2c5be WatchSource:0}: Error finding container be19a4ad89cbde2e0febdd8e218ac9be219daa5a2e8c032999d8f7494ca2c5be: Status 404 returned error can't find the container with id be19a4ad89cbde2e0febdd8e218ac9be219daa5a2e8c032999d8f7494ca2c5be
Nov 03 04:34:56.561422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:56.561337    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf" event={"ID":"3a91a2b8-3399-4b58-8cba-2ceebb206ae5","Type":"ContainerStarted","Data":"b75e2160cc6e0591f67e3804377a88d7d2b34be22776524d5ec0df4ed097a453"}
Nov 03 04:34:56.561422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:56.561378    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf" event={"ID":"3a91a2b8-3399-4b58-8cba-2ceebb206ae5","Type":"ContainerStarted","Data":"be19a4ad89cbde2e0febdd8e218ac9be219daa5a2e8c032999d8f7494ca2c5be"}
Nov 03 04:34:56.681962 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:56.681908    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf" podStartSLOduration=1.6818959530000002 podStartE2EDuration="1.681895953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:34:56.680990094 +0000 UTC m=+5587.911882513" watchObservedRunningTime="2026-11-03 04:34:56.681895953 +0000 UTC m=+5587.912788370"
Nov 03 04:34:56.742326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:34:56.742296    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:35:00.564327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:00.564288    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:35:03.770873 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:03.770829    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:35:03.771525 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:03.771491    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:35:03.854709 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:03.854663    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:35:03.854896 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:03.854734    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:35:03.879576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:03.873013    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:35:03.879576 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:03.873096    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:35:05.742222 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:05.742186    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:35:05.754860 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:05.754831    5515 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:35:06.654472 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:06.654437    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:35:20.304798 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:20.304738    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:35:20.984230 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:35:20.984181    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:35:21.679163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:21.679129    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:35:26.477928 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:26.477888    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:35:50.983662 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:35:50.983624    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:35:51.183123 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:35:51.183084    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:36:02.804096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:02.804058    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"]
Nov 03 04:36:02.809747 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:02.809721    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:36:07.372435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:07.372401    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:36:13.477385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:13.477341    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:36:17.828781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:17.828739    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf" podUID="3a91a2b8-3399-4b58-8cba-2ceebb206ae5" containerName="main" containerID="cri-o://b75e2160cc6e0591f67e3804377a88d7d2b34be22776524d5ec0df4ed097a453" gracePeriod=45
Nov 03 04:36:18.057515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.057482    5515 generic.go:356] "Generic (PLEG): container finished" podID="3a91a2b8-3399-4b58-8cba-2ceebb206ae5" containerID="b75e2160cc6e0591f67e3804377a88d7d2b34be22776524d5ec0df4ed097a453" exitCode=0
Nov 03 04:36:18.057713 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.057577    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf" event={"ID":"3a91a2b8-3399-4b58-8cba-2ceebb206ae5","Type":"ContainerDied","Data":"b75e2160cc6e0591f67e3804377a88d7d2b34be22776524d5ec0df4ed097a453"}
Nov 03 04:36:18.261444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.261420    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:36:18.342960 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.342920    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home\") pod \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") "
Nov 03 04:36:18.343122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.342983    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm\") pod \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") "
Nov 03 04:36:18.343122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343053    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache\") pod \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") "
Nov 03 04:36:18.343122 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343113    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc\" (UniqueName: \"kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc\") pod \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") "
Nov 03 04:36:18.343238 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343206    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir\") pod \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") "
Nov 03 04:36:18.343238 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343213    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home" pod "3a91a2b8-3399-4b58-8cba-2ceebb206ae5" (UID: "3a91a2b8-3399-4b58-8cba-2ceebb206ae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:36:18.343307 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343240    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs\") pod \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\" (UID: \"3a91a2b8-3399-4b58-8cba-2ceebb206ae5\") "
Nov 03 04:36:18.343581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343409    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache" pod "3a91a2b8-3399-4b58-8cba-2ceebb206ae5" (UID: "3a91a2b8-3399-4b58-8cba-2ceebb206ae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:36:18.343581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343573    5515 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-home\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:36:18.343780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343594    5515 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-model-cache\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:36:18.343780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.343636    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir" pod "3a91a2b8-3399-4b58-8cba-2ceebb206ae5" (UID: "3a91a2b8-3399-4b58-8cba-2ceebb206ae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:36:18.346029 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.346005    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc" pod "3a91a2b8-3399-4b58-8cba-2ceebb206ae5" (UID: "3a91a2b8-3399-4b58-8cba-2ceebb206ae5"). InnerVolumeSpecName "kube-api-access-nl2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:36:18.346208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.346186    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm" pod "3a91a2b8-3399-4b58-8cba-2ceebb206ae5" (UID: "3a91a2b8-3399-4b58-8cba-2ceebb206ae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 04:36:18.346268 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.346258    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs" pod "3a91a2b8-3399-4b58-8cba-2ceebb206ae5" (UID: "3a91a2b8-3399-4b58-8cba-2ceebb206ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:36:18.444500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.444379    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2hc\" (UniqueName: \"kubernetes.io/projected/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-kube-api-access-nl2hc\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:36:18.444500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.444419    5515 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tmp-dir\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:36:18.444500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.444437    5515 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-tls-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:36:18.444500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.444450    5515 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a91a2b8-3399-4b58-8cba-2ceebb206ae5-dshm\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:36:18.781779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.781679    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:36:18.781779 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.781742    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:36:18.848728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.848679    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:36:18.849118 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:18.848750    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:36:19.063948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:19.063865    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf" event={"ID":"3a91a2b8-3399-4b58-8cba-2ceebb206ae5","Type":"ContainerDied","Data":"be19a4ad89cbde2e0febdd8e218ac9be219daa5a2e8c032999d8f7494ca2c5be"}
Nov 03 04:36:19.063948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:19.063912    5515 scope.go:122] "RemoveContainer" containerID="b75e2160cc6e0591f67e3804377a88d7d2b34be22776524d5ec0df4ed097a453"
Nov 03 04:36:19.063948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:19.063915    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"
Nov 03 04:36:19.230585 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:19.230557    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"]
Nov 03 04:36:19.281227 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:19.281201    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-82587b78-kserve-5856fb54bf-ktbxf"]
Nov 03 04:36:20.655605 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:20.655575    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:36:20.894012 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:20.893965    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a91a2b8-3399-4b58-8cba-2ceebb206ae5" path="/var/lib/kubelet/pods/3a91a2b8-3399-4b58-8cba-2ceebb206ae5/volumes"
Nov 03 04:36:20.983994 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:36:20.983910    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:36:22.578355 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:22.578268    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:36:22.728363 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:22.728328    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:36:23.420171 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:23.420140    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:36:33.386400 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:33.386369    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:36:43.500038 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:43.499990    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:36:50.478607 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:50.478580    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:36:50.480810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:50.480791    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:36:50.538066 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:50.538042    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:36:50.540340 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:50.540323    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:36:50.983279 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:36:50.983241    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:36:58.012953 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:36:58.012919    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:37:14.696020 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:14.695978    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"]
Nov 03 04:37:19.696783 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:19.696737    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57" podUID="5844b997-7f93-44e1-b813-ada656a64412" containerName="payload-pre-processing" containerID="cri-o://34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846" gracePeriod=25
Nov 03 04:37:19.947046 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:19.946983    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 04:37:20.096503 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.096463    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume\") pod \"5844b997-7f93-44e1-b813-ada656a64412\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") "
Nov 03 04:37:20.096728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.096563    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946\" (UniqueName: \"kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946\") pod \"5844b997-7f93-44e1-b813-ada656a64412\" (UID: \"5844b997-7f93-44e1-b813-ada656a64412\") "
Nov 03 04:37:20.096849 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.096821    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume" pod "5844b997-7f93-44e1-b813-ada656a64412" (UID: "5844b997-7f93-44e1-b813-ada656a64412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:37:20.098761 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.098736    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946" pod "5844b997-7f93-44e1-b813-ada656a64412" (UID: "5844b997-7f93-44e1-b813-ada656a64412"). InnerVolumeSpecName "kube-api-access-rg946". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:37:20.197877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.197798    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b997-7f93-44e1-b813-ada656a64412-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:37:20.197877 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.197822    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg946\" (UniqueName: \"kubernetes.io/projected/5844b997-7f93-44e1-b813-ada656a64412-kube-api-access-rg946\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:37:20.355213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.355176    5515 generic.go:356] "Generic (PLEG): container finished" podID="5844b997-7f93-44e1-b813-ada656a64412" containerID="34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846" exitCode=0
Nov 03 04:37:20.355391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.355225    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"
Nov 03 04:37:20.355391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.355265    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57" event={"ID":"5844b997-7f93-44e1-b813-ada656a64412","Type":"ContainerDied","Data":"34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846"}
Nov 03 04:37:20.355391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.355316    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-vwq57" event={"ID":"5844b997-7f93-44e1-b813-ada656a64412","Type":"ContainerDied","Data":"5ef4241b2270c55774a7ea5575c987cb42cb9382eb86fb3520b4bc3e8c4c1896"}
Nov 03 04:37:20.355391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.355342    5515 scope.go:122] "RemoveContainer" containerID="34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846"
Nov 03 04:37:20.368629 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.368608    5515 scope.go:122] "RemoveContainer" containerID="34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846"
Nov 03 04:37:20.368960 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:37:20.368934    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846\": container with ID starting with 34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846 not found: ID does not exist" containerID="34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846"
Nov 03 04:37:20.369082 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.368965    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846"} err="failed to get container status \"34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846\": rpc error: code = NotFound desc = could not find container \"34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846\": container with ID starting with 34bedaa95751863a7ad98394e74ee28a819023c8cc4843e6f9816a7563078846 not found: ID does not exist"
Nov 03 04:37:20.522299 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.522212    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"]
Nov 03 04:37:20.573018 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.572985    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-vwq57"]
Nov 03 04:37:20.892668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:20.892636    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5844b997-7f93-44e1-b813-ada656a64412" path="/var/lib/kubelet/pods/5844b997-7f93-44e1-b813-ada656a64412/volumes"
Nov 03 04:37:20.984139 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:37:20.984103    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:37:21.007905 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:21.007869    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:37:31.986991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:31.986959    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:37:32.115621 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.115586    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-bffbfd467-9htjd"]
Nov 03 04:37:32.116047 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116030    5515 cpu_manager.go:414] "containerMap: removing container" podUID="5844b997-7f93-44e1-b813-ada656a64412" containerName="payload-pre-processing"
Nov 03 04:37:32.116107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116048    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5844b997-7f93-44e1-b813-ada656a64412" containerName="payload-pre-processing"
Nov 03 04:37:32.116107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116083    5515 cpu_manager.go:414] "containerMap: removing container" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy"
Nov 03 04:37:32.116107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116089    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy"
Nov 03 04:37:32.116107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116096    5515 cpu_manager.go:414] "containerMap: removing container" podUID="3a91a2b8-3399-4b58-8cba-2ceebb206ae5" containerName="main"
Nov 03 04:37:32.116107 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116101    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a91a2b8-3399-4b58-8cba-2ceebb206ae5" containerName="main"
Nov 03 04:37:32.116275 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116157    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="5844b997-7f93-44e1-b813-ada656a64412" containerName="payload-pre-processing"
Nov 03 04:37:32.116275 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116165    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="8badc312-6536-4dab-bec9-1843688e8bf5" containerName="istio-proxy"
Nov 03 04:37:32.116275 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.116176    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a91a2b8-3399-4b58-8cba-2ceebb206ae5" containerName="main"
Nov 03 04:37:32.119073 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.119058    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.215572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.215518    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8a4fd3-336e-41d6-be39-2593a88f7e20-plugins-config-volume\") pod \"payload-processing-bffbfd467-9htjd\" (UID: \"3f8a4fd3-336e-41d6-be39-2593a88f7e20\") " pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.215746 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.215627    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrkw\" (UniqueName: \"kubernetes.io/projected/3f8a4fd3-336e-41d6-be39-2593a88f7e20-kube-api-access-dkrkw\") pod \"payload-processing-bffbfd467-9htjd\" (UID: \"3f8a4fd3-336e-41d6-be39-2593a88f7e20\") " pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.225956 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.225922    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 04:37:32.226142 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.226028    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2h5gl\""
Nov 03 04:37:32.227453 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.227433    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-bffbfd467-9htjd"]
Nov 03 04:37:32.299275 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.299185    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"]
Nov 03 04:37:32.302624 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.302604    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.316396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.316370    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6254r\" (UniqueName: \"kubernetes.io/projected/34146d82-13f4-4800-b5a4-aa7b82ff16c9-kube-api-access-6254r\") pod \"payload-pre-processing-684cfc8c5b-x84wl\" (UID: \"34146d82-13f4-4800-b5a4-aa7b82ff16c9\") " pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.316500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.316416    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8a4fd3-336e-41d6-be39-2593a88f7e20-plugins-config-volume\") pod \"payload-processing-bffbfd467-9htjd\" (UID: \"3f8a4fd3-336e-41d6-be39-2593a88f7e20\") " pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.316500 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.316475    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34146d82-13f4-4800-b5a4-aa7b82ff16c9-plugins-config-volume\") pod \"payload-pre-processing-684cfc8c5b-x84wl\" (UID: \"34146d82-13f4-4800-b5a4-aa7b82ff16c9\") " pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.316624 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.316502    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrkw\" (UniqueName: \"kubernetes.io/projected/3f8a4fd3-336e-41d6-be39-2593a88f7e20-kube-api-access-dkrkw\") pod \"payload-processing-bffbfd467-9htjd\" (UID: \"3f8a4fd3-336e-41d6-be39-2593a88f7e20\") " pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.317167 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.317146    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8a4fd3-336e-41d6-be39-2593a88f7e20-plugins-config-volume\") pod \"payload-processing-bffbfd467-9htjd\" (UID: \"3f8a4fd3-336e-41d6-be39-2593a88f7e20\") " pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.360993 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.360962    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"]
Nov 03 04:37:32.379345 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.379320    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrkw\" (UniqueName: \"kubernetes.io/projected/3f8a4fd3-336e-41d6-be39-2593a88f7e20-kube-api-access-dkrkw\") pod \"payload-processing-bffbfd467-9htjd\" (UID: \"3f8a4fd3-336e-41d6-be39-2593a88f7e20\") " pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.417729 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.417702    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6254r\" (UniqueName: \"kubernetes.io/projected/34146d82-13f4-4800-b5a4-aa7b82ff16c9-kube-api-access-6254r\") pod \"payload-pre-processing-684cfc8c5b-x84wl\" (UID: \"34146d82-13f4-4800-b5a4-aa7b82ff16c9\") " pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.417882 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.417747    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34146d82-13f4-4800-b5a4-aa7b82ff16c9-plugins-config-volume\") pod \"payload-pre-processing-684cfc8c5b-x84wl\" (UID: \"34146d82-13f4-4800-b5a4-aa7b82ff16c9\") " pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.418409 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.418388    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34146d82-13f4-4800-b5a4-aa7b82ff16c9-plugins-config-volume\") pod \"payload-pre-processing-684cfc8c5b-x84wl\" (UID: \"34146d82-13f4-4800-b5a4-aa7b82ff16c9\") " pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.431476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.431455    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:32.480723 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.480696    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6254r\" (UniqueName: \"kubernetes.io/projected/34146d82-13f4-4800-b5a4-aa7b82ff16c9-kube-api-access-6254r\") pod \"payload-pre-processing-684cfc8c5b-x84wl\" (UID: \"34146d82-13f4-4800-b5a4-aa7b82ff16c9\") " pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.614462 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.614421    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:32.806118 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.806092    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-bffbfd467-9htjd"]
Nov 03 04:37:32.808321 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:37:32.808292    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8a4fd3_336e_41d6_be39_2593a88f7e20.slice/crio-b36dee43fd39841006f2d81239e57ecf8e3889b98df0eb7853d1aa74645bbae2 WatchSource:0}: Error finding container b36dee43fd39841006f2d81239e57ecf8e3889b98df0eb7853d1aa74645bbae2: Status 404 returned error can't find the container with id b36dee43fd39841006f2d81239e57ecf8e3889b98df0eb7853d1aa74645bbae2
Nov 03 04:37:32.985454 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:32.985433    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"]
Nov 03 04:37:32.987609 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:37:32.987583    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34146d82_13f4_4800_b5a4_aa7b82ff16c9.slice/crio-190da6db71fc0ba73cb18d29b4c09079d570d0e3866f0963280e0a0515ce692c WatchSource:0}: Error finding container 190da6db71fc0ba73cb18d29b4c09079d570d0e3866f0963280e0a0515ce692c: Status 404 returned error can't find the container with id 190da6db71fc0ba73cb18d29b4c09079d570d0e3866f0963280e0a0515ce692c
Nov 03 04:37:33.417212 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.417178    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl" event={"ID":"34146d82-13f4-4800-b5a4-aa7b82ff16c9","Type":"ContainerStarted","Data":"e479b86f3c2cbfe61637843f82ab556392f9a55a9a5247de41118971fe2886ee"}
Nov 03 04:37:33.417407 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.417323    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:33.417407 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.417354    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl" event={"ID":"34146d82-13f4-4800-b5a4-aa7b82ff16c9","Type":"ContainerStarted","Data":"190da6db71fc0ba73cb18d29b4c09079d570d0e3866f0963280e0a0515ce692c"}
Nov 03 04:37:33.418652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.418629    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-bffbfd467-9htjd" event={"ID":"3f8a4fd3-336e-41d6-be39-2593a88f7e20","Type":"ContainerStarted","Data":"c615cdaf4b846f27a725a67af7b70f51fb19ec21bda29f72adc79fa4703b4a72"}
Nov 03 04:37:33.418800 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.418668    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-bffbfd467-9htjd" event={"ID":"3f8a4fd3-336e-41d6-be39-2593a88f7e20","Type":"ContainerStarted","Data":"b36dee43fd39841006f2d81239e57ecf8e3889b98df0eb7853d1aa74645bbae2"}
Nov 03 04:37:33.418800 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.418775    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:33.536422 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.536367    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl" podStartSLOduration=1.536352504 podStartE2EDuration="1.536352504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:37:33.535243715 +0000 UTC m=+5744.766136133" watchObservedRunningTime="2026-11-03 04:37:33.536352504 +0000 UTC m=+5744.767244921"
Nov 03 04:37:33.654462 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:33.654408    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-bffbfd467-9htjd" podStartSLOduration=1.65439585 podStartE2EDuration="1.65439585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:37:33.652984934 +0000 UTC m=+5744.883877353" watchObservedRunningTime="2026-11-03 04:37:33.65439585 +0000 UTC m=+5744.885288267"
Nov 03 04:37:44.434237 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:44.434205    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-684cfc8c5b-x84wl"
Nov 03 04:37:44.434654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:44.434282    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-bffbfd467-9htjd"
Nov 03 04:37:47.416431 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:47.416389    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:37:47.929690 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:47.929652    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:37:50.558242 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:50.558207    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:37:50.985062 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:37:50.985028    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:37:52.501230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:37:52.501020    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:38:20.984109 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:38:20.984069    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:38:21.419163 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:38:21.419131    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:38:40.634097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:38:40.634060    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:38:46.915283 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:38:46.915245    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:38:50.983526 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:38:50.983484    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:39:01.185581 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:01.185512    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:39:20.983465 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:39:20.983425    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:39:21.709296 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:21.709258    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:39:27.894810 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:27.894725    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:39:28.047646 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:28.047612    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:39:28.442603 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:28.442571    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:39:30.910489 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:30.910453    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:39:42.173226 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:39:42.173189    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:39:50.983705 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:39:50.983666    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:40:01.187717 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:40:01.187683    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:40:12.686202 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:40:12.686165    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:40:20.983446 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:40:20.983407    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:40:22.115074 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:40:22.115038    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:40:27.310536 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:40:27.310502    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:40:44.179128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:40:44.179089    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:40:50.983685 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:40:50.983646    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:41:15.610113 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:15.609891    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:41:20.984583 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:41:20.984525    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:41:22.414492 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:22.414461    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:41:26.603435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:26.603397    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:41:33.784574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:33.783644    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:41:33.784574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:33.783721    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:41:33.857680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:33.857646    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:41:33.857854 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:33.857697    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:41:33.891696 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:33.891648    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:41:33.891974 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:33.891947    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:41:38.105521 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:38.105488    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:41:50.506806 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:50.506777    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:41:50.508929 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:50.508908    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:41:50.567642 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:50.567620    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:41:50.569698 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:41:50.569682    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:41:50.984071 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:41:50.984034    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:42:01.501082 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:01.501047    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:42:11.815326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:11.815294    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:42:20.611388 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:20.611352    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:42:20.985945 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:42:20.985856    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:42:22.740094 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:22.740010    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:42:29.993760 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:29.993728    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:42:38.371312 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:38.371270    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:42:47.801459 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:47.801417    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:42:51.036720 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:42:51.036680    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:42:58.013116 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:42:58.013077    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:43:15.310063 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:43:15.310031    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:43:20.983480 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:43:20.983440    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:43:23.190575 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:43:23.183245    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:43:25.354431 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:43:25.354393    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:43:50.985294 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:43:50.985246    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:43:53.390506 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:43:53.390434    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:44:16.693926 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:16.693889    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:44:20.984123 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:44:20.984085    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:44:23.474223 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:23.474182    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:27.992204 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:27.992164    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:44:46.803209 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:46.803175    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:44:48.195266 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.195205    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:48.788716 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.788677    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:44:48.789036 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.789001    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbt8" podUID="54478bf9-f992-496c-b32c-979cea64a63c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:44:48.838244 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.835718    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:48.852695 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.852665    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-nv7sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:44:48.852967 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.852941    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-nv7sc" podUID="34655d27-700c-4347-beac-dfb690ec4e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:44:48.907574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.891728    5515 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-58f56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 04:44:48.907574 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:48.891811    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-58f56" podUID="f64240ef-e871-4000-ad47-b8343dc61b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 04:44:49.147083 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:49.146997    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:50.983239 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:44:50.983199    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:44:54.192497 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:54.192471    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:54.446192 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:54.446104    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:54.601951 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:54.601911    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:57.483507 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:57.483478    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:57.659852 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:57.659810    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:44:58.054652 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.054608    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"]
Nov 03 04:44:58.058903 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.058878    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.116586 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.116538    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"]
Nov 03 04:44:58.150091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.150068    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.150217 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.150108    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9x5h\" (UniqueName: \"kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h\") pod \"payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.165442 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.165421    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bcb427\""
Nov 03 04:44:58.165442 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.165436    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-14136b17-dockercfg-wp5cf\""
Nov 03 04:44:58.251084 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.251048    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.251273 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.251096    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x5h\" (UniqueName: \"kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h\") pod \"payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.251760 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.251741    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.313596 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.313523    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9x5h\" (UniqueName: \"kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h\") pod \"payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.372723 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.372691    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:58.746925 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:58.746900    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"]
Nov 03 04:44:58.749070 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:44:58.749042    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7c0f8_80ff_4c6c_a9e2_732b086d8dc6.slice/crio-2eecc386450cf2c64c1c7ff3d0c36fcd38148dc800abc10b392439a968667872 WatchSource:0}: Error finding container 2eecc386450cf2c64c1c7ff3d0c36fcd38148dc800abc10b392439a968667872: Status 404 returned error can't find the container with id 2eecc386450cf2c64c1c7ff3d0c36fcd38148dc800abc10b392439a968667872
Nov 03 04:44:59.652036 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:59.651992    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" event={"ID":"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6","Type":"ContainerStarted","Data":"d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809"}
Nov 03 04:44:59.652213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:59.652055    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" event={"ID":"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6","Type":"ContainerStarted","Data":"2eecc386450cf2c64c1c7ff3d0c36fcd38148dc800abc10b392439a968667872"}
Nov 03 04:44:59.652213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:59.652103    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:44:59.772427 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:44:59.772373    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" podStartSLOduration=1.772358656 podStartE2EDuration="1.772358656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:44:59.771129626 +0000 UTC m=+6191.002022044" watchObservedRunningTime="2026-11-03 04:44:59.772358656 +0000 UTC m=+6191.003251074"
Nov 03 04:45:00.439298 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.439262    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"]
Nov 03 04:45:00.443005 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.442984    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.472941 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.472910    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.473055 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.472974    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.473103 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.473063    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.473219 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.473198    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftfg\" (UniqueName: \"kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.496211 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.496187    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-bcb427-serving-cert\""
Nov 03 04:45:00.548517 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.548479    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"]
Nov 03 04:45:00.574853 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.574822    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftfg\" (UniqueName: \"kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.575008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.574861    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.575008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.574910    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.575008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.574963    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.575193 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:45:00.575084    5515 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-serving-cert: secret "maas-api-e2e-worker-single-main-bcb427-serving-cert" not found
Nov 03 04:45:00.575193 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:45:00.575164    5515 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls podName:e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d nodeName:}" failed. No retries permitted until 2026-11-03 04:45:01.075140231 +0000 UTC m=+6192.306032645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls") pod "maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" (UID: "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d") : secret "maas-api-e2e-worker-single-main-bcb427-serving-cert" not found
Nov 03 04:45:00.575646 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.575623    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.578289 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.578262    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.639668 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.639606    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftfg\" (UniqueName: \"kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:00.639930 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.639858    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:00.804001 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.803902    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:00.951129 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:00.951089    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:01.080642 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:01.080534    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:01.083056 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:01.083035    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:01.272851 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:01.272809    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:45:01.355716 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:01.355639    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:01.731025 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:01.731002    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"]
Nov 03 04:45:01.733296 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:45:01.733265    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2650b4c_3ab3_4e0a_93e4_bdadb57e4f5d.slice/crio-6771d16710c6cf5b5eb099db793396d1ab46ffd476a618e1c78edd90db691d61 WatchSource:0}: Error finding container 6771d16710c6cf5b5eb099db793396d1ab46ffd476a618e1c78edd90db691d61: Status 404 returned error can't find the container with id 6771d16710c6cf5b5eb099db793396d1ab46ffd476a618e1c78edd90db691d61
Nov 03 04:45:01.734844 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:01.734821    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:45:02.666435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:02.666402    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" event={"ID":"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d","Type":"ContainerStarted","Data":"6771d16710c6cf5b5eb099db793396d1ab46ffd476a618e1c78edd90db691d61"}
Nov 03 04:45:03.673023 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:03.672987    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" event={"ID":"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d","Type":"ContainerStarted","Data":"5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b"}
Nov 03 04:45:03.673427 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:03.673085    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:03.793927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:03.793878    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" podStartSLOduration=2.7911905040000002 podStartE2EDuration="3.793865284s" totalImagesPullingTime="1.00267478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:45:03.793425314 +0000 UTC m=+6195.024317743" watchObservedRunningTime="2026-11-03 04:45:03.793865284 +0000 UTC m=+6195.024757735"
Nov 03 04:45:05.582954 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:05.582916    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:09.682794 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:09.682762    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:45:10.577179 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:10.577130    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:10.659809 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:10.659784    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:45:13.626296 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:13.626270    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:45:15.310604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:15.310527    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:19.124859 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:19.124830    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:19.750718 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:19.750673    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:19.900177 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:19.900146    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:20.051477 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:20.051392    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:20.386487 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:20.386456    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:20.983657 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:45:20.983615    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:45:23.830763 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:23.830689    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:24.153485 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:24.153456    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:25.266950 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:25.266908    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:26.715251 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:26.715218    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:28.907655 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:28.907624    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:45:30.075882 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:30.075847    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:34.807031 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:34.806999    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:39.681187 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:39.681143    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:44.390919 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:44.390880    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:49.287678 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:49.287642    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:50.983723 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:45:50.983686    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:45:53.309459 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:53.309422    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:45:54.084933 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:54.084900    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:56.039875 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:56.039844    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:45:58.885428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:45:58.885399    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:03.683414 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:03.683383    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:08.480300 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:08.480264    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:13.284748 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:13.284716    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:18.088079 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:18.088042    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:19.414663 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:19.414623    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:19.872019 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:19.871984    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:20.983392 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:46:20.983346    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:46:22.765609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:22.765564    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:46:22.885692 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:22.885660    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:24.192986 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:24.192946    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:25.027719 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:25.027683    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:25.501526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:25.501493    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"]
Nov 03 04:46:25.867819 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:25.867786    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:26.017467 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:26.017431    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:26.353920 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:26.353835    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:27.136589 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:27.136558    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"]
Nov 03 04:46:27.137051 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:27.136784    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" podUID="e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" containerName="maas-api" containerID="cri-o://5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b" gracePeriod=30
Nov 03 04:46:27.236614 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:27.236583    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:28.376566 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.376520    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:46:28.478091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.478019    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs\") pod \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") "
Nov 03 04:46:28.478091 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.478080    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca\") pod \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") "
Nov 03 04:46:28.478281 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.478150    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg\" (UniqueName: \"kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg\") pod \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") "
Nov 03 04:46:28.478281 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.478173    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\") pod \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\" (UID: \"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d\") "
Nov 03 04:46:28.478513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.478491    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca" pod "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" (UID: "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:46:28.480424 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.480392    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls" pod "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" (UID: "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:46:28.480528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.480426    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs" pod "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" (UID: "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:46:28.480528 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.480428    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg" pod "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" (UID: "e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d"). InnerVolumeSpecName "kube-api-access-nftfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:46:28.579032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.579002    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:46:28.579032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.579028    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:46:28.579223 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.579038    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nftfg\" (UniqueName: \"kubernetes.io/projected/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-kube-api-access-nftfg\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:46:28.579223 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:28.579050    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:46:29.076504 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.076466    5515 generic.go:356] "Generic (PLEG): container finished" podID="e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" containerID="5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b" exitCode=0
Nov 03 04:46:29.076722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.076514    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"
Nov 03 04:46:29.076722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.076543    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" event={"ID":"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d","Type":"ContainerDied","Data":"5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b"}
Nov 03 04:46:29.076722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.076618    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8" event={"ID":"e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d","Type":"ContainerDied","Data":"6771d16710c6cf5b5eb099db793396d1ab46ffd476a618e1c78edd90db691d61"}
Nov 03 04:46:29.076722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.076639    5515 scope.go:122] "RemoveContainer" containerID="5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b"
Nov 03 04:46:29.087790 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.087769    5515 scope.go:122] "RemoveContainer" containerID="5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b"
Nov 03 04:46:29.088122 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:46:29.088100    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b\": container with ID starting with 5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b not found: ID does not exist" containerID="5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b"
Nov 03 04:46:29.088186 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.088127    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b"} err="failed to get container status \"5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b\": rpc error: code = NotFound desc = could not find container \"5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b\": container with ID starting with 5454f2170f38e59bfaf5820012835405e6bb344a3ec89681c3d7458b4646750b not found: ID does not exist"
Nov 03 04:46:29.244931 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.244857    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"]
Nov 03 04:46:29.252785 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:29.252753    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcb427-7c68bfbd5c-z4ht8"]
Nov 03 04:46:30.501994 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:30.501950    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing" containerID="cri-o://d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809" gracePeriod=25
Nov 03 04:46:30.658725 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:30.658685    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:46:30.892039 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:30.892001    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" path="/var/lib/kubelet/pods/e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d/volumes"
Nov 03 04:46:32.292784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:32.292741    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:38.111618 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:38.111586    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:38.259979 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:38.259944    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:38.564233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:38.564151    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:46:40.658781 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:40.658748    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 04:46:49.556115 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:49.556073    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:46:50.535973 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.535954    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:46:50.537973 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.537956    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:46:50.598414 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.598393    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:46:50.600732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.600716    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:46:50.658184 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:46:50.658160    5515 logging.go:55] [core] [Channel #14669 SubChannel #14670]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.241:50052", ServerName: "10.134.0.241:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.241:50052: connect: connection refused"
Nov 03 04:46:50.753413 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.753388    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:46:50.883265 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.883242    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h\" (UniqueName: \"kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h\") pod \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") "
Nov 03 04:46:50.883438 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.883302    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume\") pod \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\" (UID: \"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6\") "
Nov 03 04:46:50.883734 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.883711    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume" pod "aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" (UID: "aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:46:50.885609 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.885590    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h" pod "aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" (UID: "aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6"). InnerVolumeSpecName "kube-api-access-v9x5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:46:50.984328 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:46:50.984291    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:46:50.984665 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.984637    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9x5h\" (UniqueName: \"kubernetes.io/projected/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-kube-api-access-v9x5h\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:46:50.984665 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:50.984666    5515 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6-plugins-config-volume\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:46:51.180612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.180487    5515 generic.go:356] "Generic (PLEG): container finished" podID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerID="d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809" exitCode=0
Nov 03 04:46:51.180612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.180539    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"
Nov 03 04:46:51.180612 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.180577    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" event={"ID":"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6","Type":"ContainerDied","Data":"d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809"}
Nov 03 04:46:51.180888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.180640    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" event={"ID":"aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6","Type":"ContainerDied","Data":"2eecc386450cf2c64c1c7ff3d0c36fcd38148dc800abc10b392439a968667872"}
Nov 03 04:46:51.180888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.180670    5515 scope.go:122] "RemoveContainer" containerID="d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809"
Nov 03 04:46:51.189395 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.189376    5515 scope.go:122] "RemoveContainer" containerID="d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809"
Nov 03 04:46:51.189682 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:46:51.189660    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809\": container with ID starting with d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809 not found: ID does not exist" containerID="d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809"
Nov 03 04:46:51.189773 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.189689    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809"} err="failed to get container status \"d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809\": rpc error: code = NotFound desc = could not find container \"d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809\": container with ID starting with d9cbd287226ba97fe067713a67d45dbdc8c35d86d9ca328acaec619a8feb4809 not found: ID does not exist"
Nov 03 04:46:51.347526 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.347492    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"]
Nov 03 04:46:51.353359 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.353334    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg"]
Nov 03 04:46:51.658539 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:51.658503    5515 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bcb427-59f747597g7lfg" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.241:50052\" within 1s: context deadline exceeded"
Nov 03 04:46:51.658957 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:46:51.658586    5515 logging.go:55] [core] [Channel #14669 SubChannel #14670]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.241:50052", ServerName: "10.134.0.241:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.241:50052: operation was canceled"
Nov 03 04:46:52.892688 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:46:52.892612    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" path="/var/lib/kubelet/pods/aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6/volumes"
Nov 03 04:47:01.610949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:01.610918    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:01.751476 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:01.751445    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:03.851948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:03.851909    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:04.592917 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:04.592884    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:07.773505 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:07.773462    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:07.915189 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:07.915157    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:20.984000 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:47:20.983960    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:47:24.536165 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:47:24.536135    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:47:50.983766 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:47:50.983729    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:48:20.982970 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:48:20.982923    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:48:24.933477 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:48:24.933405    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:48:49.113435 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:48:49.113399    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:48:51.048765 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:48:51.048724    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:49:06.605317 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:06.605284    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:49:16.410780 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:16.410746    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:49:20.983345 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:49:20.983312    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:49:25.083064 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:25.083029    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:49:25.242263 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:25.242236    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:49:25.386893 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:25.386862    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:49:28.786071 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786037    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"]
Nov 03 04:49:28.786486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786450    5515 cpu_manager.go:414] "containerMap: removing container" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing"
Nov 03 04:49:28.786486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786461    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing"
Nov 03 04:49:28.786486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786484    5515 cpu_manager.go:414] "containerMap: removing container" podUID="e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" containerName="maas-api"
Nov 03 04:49:28.786654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786489    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" containerName="maas-api"
Nov 03 04:49:28.786654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786571    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff7c0f8-80ff-4c6c-a9e2-732b086d8dc6" containerName="payload-processing"
Nov 03 04:49:28.786654 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.786586    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2650b4c-3ab3-4e0a-93e4-bdadb57e4f5d" containerName="maas-api"
Nov 03 04:49:28.789774 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.789758    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.867410 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.867373    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5bq\" (UniqueName: \"kubernetes.io/projected/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-kube-api-access-hk5bq\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.867410 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.867413    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-maas-api-tls\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.867628 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.867491    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-metrics-certs\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.867628 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.867514    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-openshift-service-ca\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.895842 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.895812    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"]
Nov 03 04:49:28.968429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.968397    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5bq\" (UniqueName: \"kubernetes.io/projected/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-kube-api-access-hk5bq\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.968429 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.968429    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-maas-api-tls\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.968725 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.968464    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-metrics-certs\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.968725 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.968478    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-openshift-service-ca\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.969513 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.969489    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-openshift-service-ca\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.971137 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.971108    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-maas-api-tls\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:28.971250 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:28.971148    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-metrics-certs\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:29.030470 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:29.030437    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5bq\" (UniqueName: \"kubernetes.io/projected/b3fdc7bb-0ddd-45da-a57f-99016dc307d0-kube-api-access-hk5bq\") pod \"maas-api-b77bddc5b-8xws6\" (UID: \"b3fdc7bb-0ddd-45da-a57f-99016dc307d0\") " pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:29.101728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:29.101671    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:29.471154 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:29.471130    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"]
Nov 03 04:49:29.472996 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:49:29.472966    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fdc7bb_0ddd_45da_a57f_99016dc307d0.slice/crio-248009ca1f9726f80b2f02e5b418b4aaafafd9226fa05b555bf4b7c8779f9c00 WatchSource:0}: Error finding container 248009ca1f9726f80b2f02e5b418b4aaafafd9226fa05b555bf4b7c8779f9c00: Status 404 returned error can't find the container with id 248009ca1f9726f80b2f02e5b418b4aaafafd9226fa05b555bf4b7c8779f9c00
Nov 03 04:49:29.920197 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:29.920161    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6" event={"ID":"b3fdc7bb-0ddd-45da-a57f-99016dc307d0","Type":"ContainerStarted","Data":"248009ca1f9726f80b2f02e5b418b4aaafafd9226fa05b555bf4b7c8779f9c00"}
Nov 03 04:49:30.926330 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:30.926293    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6" event={"ID":"b3fdc7bb-0ddd-45da-a57f-99016dc307d0","Type":"ContainerStarted","Data":"cb942362157b22effc8f2eb35fb4505bb3653b6d022cc91104e1dde530cc5607"}
Nov 03 04:49:30.926833 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:30.926411    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:31.046144 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:31.046094    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6" podStartSLOduration=2.192386007 podStartE2EDuration="3.046081799s" totalImagesPullingTime="853.695792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:49:31.045848816 +0000 UTC m=+6462.276741234" watchObservedRunningTime="2026-11-03 04:49:31.046081799 +0000 UTC m=+6462.276974215"
Nov 03 04:49:32.821439 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:32.821403    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:49:36.936028 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:36.935990    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b77bddc5b-8xws6"
Nov 03 04:49:37.108396 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:37.108363    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"]
Nov 03 04:49:37.108666 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:37.108629    5515 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452" podUID="7332092d-faf6-464b-a496-e4a906ee83af" containerName="maas-api" containerID="cri-o://436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e" gracePeriod=30
Nov 03 04:49:38.369972 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.369947    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 04:49:38.456404 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.456342    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7\" (UniqueName: \"kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7\") pod \"7332092d-faf6-464b-a496-e4a906ee83af\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") "
Nov 03 04:49:38.456404 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.456382    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca\") pod \"7332092d-faf6-464b-a496-e4a906ee83af\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") "
Nov 03 04:49:38.456649 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.456491    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs\") pod \"7332092d-faf6-464b-a496-e4a906ee83af\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") "
Nov 03 04:49:38.456649 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.456528    5515 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls\") pod \"7332092d-faf6-464b-a496-e4a906ee83af\" (UID: \"7332092d-faf6-464b-a496-e4a906ee83af\") "
Nov 03 04:49:38.456830 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.456809    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca" pod "7332092d-faf6-464b-a496-e4a906ee83af" (UID: "7332092d-faf6-464b-a496-e4a906ee83af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 04:49:38.458653 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.458628    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls" pod "7332092d-faf6-464b-a496-e4a906ee83af" (UID: "7332092d-faf6-464b-a496-e4a906ee83af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:49:38.458782 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.458716    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7" pod "7332092d-faf6-464b-a496-e4a906ee83af" (UID: "7332092d-faf6-464b-a496-e4a906ee83af"). InnerVolumeSpecName "kube-api-access-ng8s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 04:49:38.458782 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.458747    5515 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs" pod "7332092d-faf6-464b-a496-e4a906ee83af" (UID: "7332092d-faf6-464b-a496-e4a906ee83af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 04:49:38.557311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.557280    5515 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-maas-api-tls\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:49:38.557311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.557307    5515 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8s7\" (UniqueName: \"kubernetes.io/projected/7332092d-faf6-464b-a496-e4a906ee83af-kube-api-access-ng8s7\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:49:38.557311 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.557317    5515 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7332092d-faf6-464b-a496-e4a906ee83af-openshift-service-ca\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:49:38.557532 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.557326    5515 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7332092d-faf6-464b-a496-e4a906ee83af-metrics-certs\") on node \"ip-10-0-138-11.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 04:49:38.960677 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.960639    5515 generic.go:356] "Generic (PLEG): container finished" podID="7332092d-faf6-464b-a496-e4a906ee83af" containerID="436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e" exitCode=0
Nov 03 04:49:38.960822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.960699    5515 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"
Nov 03 04:49:38.960822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.960725    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452" event={"ID":"7332092d-faf6-464b-a496-e4a906ee83af","Type":"ContainerDied","Data":"436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e"}
Nov 03 04:49:38.960822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.960791    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452" event={"ID":"7332092d-faf6-464b-a496-e4a906ee83af","Type":"ContainerDied","Data":"945bca5b26ef1f73ccdbcbeeb286df62324aaa92e580aacee81636a5d3101e3d"}
Nov 03 04:49:38.960822 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.960819    5515 scope.go:122] "RemoveContainer" containerID="436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e"
Nov 03 04:49:38.969899 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.969879    5515 scope.go:122] "RemoveContainer" containerID="436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e"
Nov 03 04:49:38.970130 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:49:38.970110    5515 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e\": container with ID starting with 436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e not found: ID does not exist" containerID="436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e"
Nov 03 04:49:38.970189 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:38.970137    5515 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e"} err="failed to get container status \"436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e\": rpc error: code = NotFound desc = could not find container \"436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e\": container with ID starting with 436c1c9c51b3d3c79e90a8b92a12aaf9c0836398dc4d07aec1cd70629b2a2f1e not found: ID does not exist"
Nov 03 04:49:39.126645 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:39.126614    5515 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"]
Nov 03 04:49:39.177356 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:39.177289    5515 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55cd9fdc56-bg452"]
Nov 03 04:49:40.892857 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:40.892815    5515 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7332092d-faf6-464b-a496-e4a906ee83af" path="/var/lib/kubelet/pods/7332092d-faf6-464b-a496-e4a906ee83af/volumes"
Nov 03 04:49:41.677143 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:49:41.677113    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:49:50.983203 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:49:50.983168    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:50:17.777593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:50:17.777490    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:50:20.984607 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:50:20.984568    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:50:25.604577 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:50:25.604131    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:50:30.663854 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:50:30.663825    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:50:50.982687 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:50:50.982650    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:50:53.503764 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:50:53.503733    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:51:06.488922 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:06.488890    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:51:20.984730 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:51:20.984685    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:51:25.946719 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:25.946624    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:51:38.618712 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:38.618686    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:51:49.292537 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:49.292498    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:51:50.576583 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:50.576534    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:51:50.600659 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:50.600633    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:51:50.717104 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:50.717078    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:51:50.731812 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:51:50.731784    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:51:51.052925 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:51:51.052884    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:52:02.718783 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:52:02.718748    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:52:14.498991 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:52:14.498960    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:52:20.982860 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:52:20.982821    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:52:26.277731 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:52:26.277694    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:52:38.010097 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:52:38.010062    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:52:48.799008 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:52:48.798972    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:52:50.986215 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:52:50.986175    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:52:57.376466 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:52:57.376387    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:53:18.378096 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:53:18.378060    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:53:21.051826 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:53:21.051783    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:53:26.707327 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:53:26.707289    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:53:42.211391 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:53:42.211360    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:53:50.983059 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:53:50.983018    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:54:20.983720 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:54:20.983679    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:54:28.371883 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:54:28.371641    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:54:50.983433 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:54:50.983396    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:55:20.986338 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:55:20.986282    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:55:27.358737 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:55:27.358694    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:55:39.288449 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:55:39.288410    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lhs8n"]
Nov 03 04:55:50.983071 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:55:50.983032    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:55:55.616818 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:55:55.616722    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:56:12.080234 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:12.080195    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d99f9cf84-h9p42_93ba1b83-66ba-4138-ae86-98b9dd588eed/manager/0.log"
Nov 03 04:56:12.451856 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:12.451831    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7tdsv_108ea924-255e-45c4-85a3-deaa09ddb1f0/manager/0.log"
Nov 03 04:56:13.650010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:13.649981    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7957588c98-phcw7_5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e/manager/0.log"
Nov 03 04:56:14.175680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:14.175649    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-discovery-845d48d7b4-6k262_7ccb2946-a5ce-4a22-b5eb-301c6373c834/maas-discovery/0.log"
Nov 03 04:56:16.039297 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:16.039267    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b77bddc5b-8xws6_b3fdc7bb-0ddd-45da-a57f-99016dc307d0/maas-api/0.log"
Nov 03 04:56:20.982885 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:56:20.982838    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:56:23.247208 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:23.247181    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-m5qlx_5c3cc519-a61c-4b59-905e-6c06aa56da13/manager/0.log"
Nov 03 04:56:23.477902 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:23.477874    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sj294_0f8f7050-3fb9-4229-b35a-b5d831cdb957/kuadrant-console-plugin/0.log"
Nov 03 04:56:24.296632 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:24.296603    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhs8n_83d03a51-825c-477b-9fcc-e5af9421d387/limitador/0.log"
Nov 03 04:56:25.810032 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:25.810002    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zb5pl_0f392881-5978-44e2-8284-ff5d07d80563/discovery/0.log"
Nov 03 04:56:26.414732 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:26.414702    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5cd8cf48-sz5c2_4e3fdaae-993d-42e5-85fd-5bcdf96e67af/kube-auth-proxy/0.log"
Nov 03 04:56:26.652917 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:26.652887    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-4nbt8_54478bf9-f992-496c-b32c-979cea64a63c/istio-proxy/0.log"
Nov 03 04:56:26.936170 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:26.936145    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-58f56_f64240ef-e871-4000-ad47-b8343dc61b53/istio-proxy/0.log"
Nov 03 04:56:27.703040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:27.702997    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:56:28.326830 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:28.326804    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-nv7sc_34655d27-700c-4347-beac-dfb690ec4e7b/istio-proxy/0.log"
Nov 03 04:56:29.647392 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:29.647359    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-684cfc8c5b-x84wl_34146d82-13f4-4800-b5a4-aa7b82ff16c9/payload-pre-processing/0.log"
Nov 03 04:56:29.877439 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:29.877402    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-bffbfd467-9htjd_3f8a4fd3-336e-41d6-be39-2593a88f7e20/payload-processing/0.log"
Nov 03 04:56:31.538106 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:31.538078    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-p8clw_d373288c-3e8d-4c1d-b7a7-90eb5dd2e488/main/0.log"
Nov 03 04:56:31.810838 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:31.810765    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-jpqzk_79d93c40-3f2d-41eb-8006-2493304b0e38/main/0.log"
Nov 03 04:56:32.660722 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:32.660694    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-d45dp_ef1853a0-c729-463f-ade2-76de6dddca26/main/0.log"
Nov 03 04:56:33.205530 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:33.205500    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-9vwlw_e1aa2223-e7d4-4647-8be5-402eb71fed01/main/0.log"
Nov 03 04:56:44.096795 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.096756    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:56:44.222846 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.222814    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ljm7/must-gather-k4xwd"]
Nov 03 04:56:44.223385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.223356    5515 cpu_manager.go:414] "containerMap: removing container" podUID="7332092d-faf6-464b-a496-e4a906ee83af" containerName="maas-api"
Nov 03 04:56:44.223385 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.223382    5515 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7332092d-faf6-464b-a496-e4a906ee83af" containerName="maas-api"
Nov 03 04:56:44.223584 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.223533    5515 memory_manager.go:365] "RemoveStaleState removing state" podUID="7332092d-faf6-464b-a496-e4a906ee83af" containerName="maas-api"
Nov 03 04:56:44.226927 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.226910    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.252948 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.252930    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:56:44.284474 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.284449    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/must-gather-k4xwd"]
Nov 03 04:56:44.285444 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.285429    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ljm7\"/\"kube-root-ca.crt\""
Nov 03 04:56:44.333040 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.333020    5515 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ljm7\"/\"openshift-service-ca.crt\""
Nov 03 04:56:44.333138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.333052    5515 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ljm7\"/\"default-dockercfg-w89j4\""
Nov 03 04:56:44.335871 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.335850    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cdc61c88-ce01-4376-9cca-f24530ae98de-must-gather-output\") pod \"must-gather-k4xwd\" (UID: \"cdc61c88-ce01-4376-9cca-f24530ae98de\") " pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.335937 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.335918    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7hs\" (UniqueName: \"kubernetes.io/projected/cdc61c88-ce01-4376-9cca-f24530ae98de-kube-api-access-gt7hs\") pod \"must-gather-k4xwd\" (UID: \"cdc61c88-ce01-4376-9cca-f24530ae98de\") " pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.436776 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.436754    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cdc61c88-ce01-4376-9cca-f24530ae98de-must-gather-output\") pod \"must-gather-k4xwd\" (UID: \"cdc61c88-ce01-4376-9cca-f24530ae98de\") " pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.436910 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.436807    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7hs\" (UniqueName: \"kubernetes.io/projected/cdc61c88-ce01-4376-9cca-f24530ae98de-kube-api-access-gt7hs\") pod \"must-gather-k4xwd\" (UID: \"cdc61c88-ce01-4376-9cca-f24530ae98de\") " pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.437086 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.437068    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cdc61c88-ce01-4376-9cca-f24530ae98de-must-gather-output\") pod \"must-gather-k4xwd\" (UID: \"cdc61c88-ce01-4376-9cca-f24530ae98de\") " pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.498278 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.498253    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7hs\" (UniqueName: \"kubernetes.io/projected/cdc61c88-ce01-4376-9cca-f24530ae98de-kube-api-access-gt7hs\") pod \"must-gather-k4xwd\" (UID: \"cdc61c88-ce01-4376-9cca-f24530ae98de\") " pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.539906 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.539881    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/must-gather-k4xwd"
Nov 03 04:56:44.909326 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.909299    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/must-gather-k4xwd"]
Nov 03 04:56:44.912869 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:56:44.912844    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc61c88_ce01_4376_9cca_f24530ae98de.slice/crio-57a60e2aaa8472ab3498cc7da39a4270972559f0adf6b3283a6b8e48153f5aab WatchSource:0}: Error finding container 57a60e2aaa8472ab3498cc7da39a4270972559f0adf6b3283a6b8e48153f5aab: Status 404 returned error can't find the container with id 57a60e2aaa8472ab3498cc7da39a4270972559f0adf6b3283a6b8e48153f5aab
Nov 03 04:56:44.914619 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:44.914601    5515 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 04:56:45.272754 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:45.272671    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/must-gather-k4xwd" event={"ID":"cdc61c88-ce01-4376-9cca-f24530ae98de","Type":"ContainerStarted","Data":"57a60e2aaa8472ab3498cc7da39a4270972559f0adf6b3283a6b8e48153f5aab"}
Nov 03 04:56:46.281253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:46.281172    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/must-gather-k4xwd" event={"ID":"cdc61c88-ce01-4376-9cca-f24530ae98de","Type":"ContainerStarted","Data":"1e851144e1ee25dd46c364bf6114a37413d4d3c32285d5f7c935d1dacfeb04e3"}
Nov 03 04:56:46.281253 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:46.281219    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/must-gather-k4xwd" event={"ID":"cdc61c88-ce01-4376-9cca-f24530ae98de","Type":"ContainerStarted","Data":"f1b3408d390d5345fa50949941120bbc15693471cd275b755a8ad3697b8969ab"}
Nov 03 04:56:46.399392 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:46.399346    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4ljm7/must-gather-k4xwd" podStartSLOduration=1.320774211 podStartE2EDuration="2.399333097s" totalImagesPullingTime="1.078558886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 04:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:56:46.397942602 +0000 UTC m=+6897.628835018" watchObservedRunningTime="2026-11-03 04:56:46.399333097 +0000 UTC m=+6897.630225535"
Nov 03 04:56:50.677735 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:50.677685    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:56:50.680162 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:50.680142    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:56:50.766885 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:50.766863    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jb7sk_44942777-b4b6-4264-aeff-7c88943cf14e/kube-multus/0.log"
Nov 03 04:56:50.769227 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:50.769205    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cvkw5_a3c603e8-5a0a-42af-a866-3439e1fdef56/ovn-acl-logging/0.log"
Nov 03 04:56:50.984520 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:56:50.984436    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:56:54.026601 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:54.022142    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rqm5c_5000ce1d-93ee-4bed-bf15-7ecb2d1301e9/global-pull-secret-syncer/0.log"
Nov 03 04:56:55.289428 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:55.289387    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d99f9cf84-h9p42_93ba1b83-66ba-4138-ae86-98b9dd588eed/manager/0.log"
Nov 03 04:56:55.480897 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:55.480769    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7tdsv_108ea924-255e-45c4-85a3-deaa09ddb1f0/manager/0.log"
Nov 03 04:56:56.018579 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:56.015030    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log"
Nov 03 04:56:56.494230 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:56.494198    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7957588c98-phcw7_5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e/manager/0.log"
Nov 03 04:56:56.672335 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:56.672305    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-discovery-845d48d7b4-6k262_7ccb2946-a5ce-4a22-b5eb-301c6373c834/maas-discovery/0.log"
Nov 03 04:56:57.637424 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:56:57.637392    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-11.us-west-1.compute.internal_8452f8e9cc16a665620f2f318ca698f0/haproxy/0.log"
Nov 03 04:57:00.030044 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:00.030007    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:57:00.182830 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:00.182790    5515 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xflz_35125e2e-9a2d-4577-98ca-5966cfe1d3a1/konnectivity-agent/0.log" limit=1024
Nov 03 04:57:01.026756 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:01.026724    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d99f9cf84-h9p42_93ba1b83-66ba-4138-ae86-98b9dd588eed/manager/0.log"
Nov 03 04:57:01.220680 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:01.220616    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7tdsv_108ea924-255e-45c4-85a3-deaa09ddb1f0/manager/0.log"
Nov 03 04:57:02.584104 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:02.584074    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7957588c98-phcw7_5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e/manager/0.log"
Nov 03 04:57:02.822066 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:02.822020    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-discovery-845d48d7b4-6k262_7ccb2946-a5ce-4a22-b5eb-301c6373c834/maas-discovery/0.log"
Nov 03 04:57:02.897456 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:02.897409    5515 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"]
Nov 03 04:57:02.906158 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:02.906125    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:02.993233 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:02.993199    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"]
Nov 03 04:57:03.031888 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.031846    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-podres\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.032213 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.032192    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-proc\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.032351 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.032337    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-sys\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.032604 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.032584    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cgt\" (UniqueName: \"kubernetes.io/projected/9d3235c0-4b3f-4fac-b230-4323896c935b-kube-api-access-22cgt\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.032786 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.032772    5515 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-lib-modules\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134741 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134708    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-lib-modules\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134793    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-podres\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134820    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-proc\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134841    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-sys\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134871    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-lib-modules\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134891    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-podres\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134905    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-proc\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134911    5515 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cgt\" (UniqueName: \"kubernetes.io/projected/9d3235c0-4b3f-4fac-b230-4323896c935b-kube-api-access-22cgt\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.134949 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.134944    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d3235c0-4b3f-4fac-b230-4323896c935b-sys\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.198119 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.198040    5515 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cgt\" (UniqueName: \"kubernetes.io/projected/9d3235c0-4b3f-4fac-b230-4323896c935b-kube-api-access-22cgt\") pod \"perf-node-gather-daemonset-mjmrf\" (UID: \"9d3235c0-4b3f-4fac-b230-4323896c935b\") " pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.227515 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.227485    5515 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:03.442316 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.442273    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-m5qlx_5c3cc519-a61c-4b59-905e-6c06aa56da13/manager/0.log"
Nov 03 04:57:03.609138 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.609111    5515 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"]
Nov 03 04:57:03.611419 ip-10-0-138-11 kubenswrapper[5515]: W1103 04:57:03.611394    5515 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d3235c0_4b3f_4fac_b230_4323896c935b.slice/crio-9388779b75e76bf89c5c6b069c47d125647e2847e1891752511a084e39041a85 WatchSource:0}: Error finding container 9388779b75e76bf89c5c6b069c47d125647e2847e1891752511a084e39041a85: Status 404 returned error can't find the container with id 9388779b75e76bf89c5c6b069c47d125647e2847e1891752511a084e39041a85
Nov 03 04:57:03.856989 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:03.856957    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sj294_0f8f7050-3fb9-4229-b35a-b5d831cdb957/kuadrant-console-plugin/0.log"
Nov 03 04:57:04.406278 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:04.406234    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf" event={"ID":"9d3235c0-4b3f-4fac-b230-4323896c935b","Type":"ContainerStarted","Data":"1d4fb5409e380fb31124adbe47d1436cd730c4ec6c93f768b8820b76d759c2d9"}
Nov 03 04:57:04.406572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:04.406286    5515 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf" event={"ID":"9d3235c0-4b3f-4fac-b230-4323896c935b","Type":"ContainerStarted","Data":"9388779b75e76bf89c5c6b069c47d125647e2847e1891752511a084e39041a85"}
Nov 03 04:57:04.406572 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:04.406380    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:04.525464 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:04.525414    5515 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf" podStartSLOduration=2.52539926 podStartE2EDuration="2.52539926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 04:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 04:57:04.524529953 +0000 UTC m=+6915.755422370" watchObservedRunningTime="2026-11-03 04:57:04.52539926 +0000 UTC m=+6915.756291706"
Nov 03 04:57:04.690950 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:04.690863    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhs8n_83d03a51-825c-477b-9fcc-e5af9421d387/limitador/0.log"
Nov 03 04:57:07.297840 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:07.297813    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6d99f9cf84-h9p42_93ba1b83-66ba-4138-ae86-98b9dd588eed/manager/0.log"
Nov 03 04:57:07.491987 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:07.491960    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7tdsv_108ea924-255e-45c4-85a3-deaa09ddb1f0/manager/0.log"
Nov 03 04:57:07.723350 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:07.723321    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/alertmanager/0.log"
Nov 03 04:57:08.148010 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:08.147986    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/config-reloader/0.log"
Nov 03 04:57:08.483811 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:08.483729    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7957588c98-phcw7_5d7483a5-8b8c-4890-8a6b-9f4a3aa8d08e/manager/0.log"
Nov 03 04:57:08.582189 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:08.582151    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/kube-rbac-proxy-web/0.log"
Nov 03 04:57:08.665424 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:08.665385    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-discovery-845d48d7b4-6k262_7ccb2946-a5ce-4a22-b5eb-301c6373c834/maas-discovery/0.log"
Nov 03 04:57:08.996769 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:08.996738    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/kube-rbac-proxy/0.log"
Nov 03 04:57:09.411622 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:09.411598    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/kube-rbac-proxy-metric/0.log"
Nov 03 04:57:09.827970 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:09.827896    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/prom-label-proxy/0.log"
Nov 03 04:57:10.244775 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:10.244743    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_860635a9-c37a-4f31-a4c3-b7d2a4ed5a20/init-config-reloader/0.log"
Nov 03 04:57:10.424728 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:10.424700    5515 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4ljm7/perf-node-gather-daemonset-mjmrf"
Nov 03 04:57:12.322159 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:12.322132    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-875cc949b-lpm9j_dcff55ad-f5a9-4f59-afd0-ecae4aefa472/metrics-server/0.log"
Nov 03 04:57:15.507938 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:15.507906    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zb5pl_0f392881-5978-44e2-8284-ff5d07d80563/discovery/0.log"
Nov 03 04:57:15.678645 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:15.678617    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwq7p_9e897c78-ce55-4d3f-bcfd-148c8faf01f2/node-exporter/0.log"
Nov 03 04:57:15.838784 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:15.838707    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d5cd8cf48-sz5c2_4e3fdaae-993d-42e5-85fd-5bcdf96e67af/kube-auth-proxy/0.log"
Nov 03 04:57:16.057932 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:16.057898    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-4nbt8_54478bf9-f992-496c-b32c-979cea64a63c/istio-proxy/0.log"
Nov 03 04:57:16.091699 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:16.091630    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwq7p_9e897c78-ce55-4d3f-bcfd-148c8faf01f2/kube-rbac-proxy/0.log"
Nov 03 04:57:16.272593 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:16.272537    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-58f56_f64240ef-e871-4000-ad47-b8343dc61b53/istio-proxy/0.log"
Nov 03 04:57:16.507228 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:16.507202    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwq7p_9e897c78-ce55-4d3f-bcfd-148c8faf01f2/init-textfile/0.log"
Nov 03 04:57:16.923486 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:16.923449    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qj4mb_150b0a73-6cfc-438d-853f-3302f2232ec2/kube-rbac-proxy-main/0.log"
Nov 03 04:57:17.342918 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:17.342844    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qj4mb_150b0a73-6cfc-438d-853f-3302f2232ec2/kube-rbac-proxy-self/0.log"
Nov 03 04:57:17.352767 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:17.352746    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-nv7sc_34655d27-700c-4347-beac-dfb690ec4e7b/istio-proxy/0.log"
Nov 03 04:57:17.759128 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:17.759099    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qj4mb_150b0a73-6cfc-438d-853f-3302f2232ec2/openshift-state-metrics/0.log"
Nov 03 04:57:18.423219 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:18.423184    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-684cfc8c5b-x84wl_34146d82-13f4-4800-b5a4-aa7b82ff16c9/payload-pre-processing/0.log"
Nov 03 04:57:18.636517 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:18.636481    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-bffbfd467-9htjd_3f8a4fd3-336e-41d6-be39-2593a88f7e20/payload-processing/0.log"
Nov 03 04:57:20.986513 ip-10-0-138-11 kubenswrapper[5515]: E1103 04:57:20.986473    5515 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hgj4z" containerName="extractor"
Nov 03 04:57:22.351598 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:22.351569    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766f4dd556-x6pjk_e5baba82-91af-4032-832a-2c12ac90ae0c/telemeter-client/0.log"
Nov 03 04:57:22.765144 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:22.765108    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766f4dd556-x6pjk_e5baba82-91af-4032-832a-2c12ac90ae0c/reload/0.log"
Nov 03 04:57:23.178669 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:23.178645    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766f4dd556-x6pjk_e5baba82-91af-4032-832a-2c12ac90ae0c/kube-rbac-proxy/0.log"
Nov 03 04:57:24.459910 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:24.459885    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-m5qlx_5c3cc519-a61c-4b59-905e-6c06aa56da13/manager/0.log"
Nov 03 04:57:24.672370 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:24.672341    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sj294_0f8f7050-3fb9-4229-b35a-b5d831cdb957/kuadrant-console-plugin/0.log"
Nov 03 04:57:25.101698 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:25.101668    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lhs8n_83d03a51-825c-477b-9fcc-e5af9421d387/limitador/0.log"
Nov 03 04:57:27.956389 ip-10-0-138-11 kubenswrapper[5515]: I1103 04:57:27.956351    5515 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zb5pl_0f392881-5978-44e2-8284-ff5d07d80563/discovery/0.log"